File Storage Question

In a web farm environment, how to store the user uploading files for future
access?
Store them in db, local file system or centralized file server?

If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file system?
or the centralized
file server?

Should I create only one folder or lots of subfolders to organize the file
structure for better access
from file system?

Re: File Storage Question

Depends on what you want to do with the files and what they represent. Since
you haven't said, ASP.NET uploads files into server memory, buffers it and
then writes it to disk in the simplest case. Where you choose to store it
depends on how frequently you want to access it and the responsiveness you
want from the application, among other things.

--
--
Regards,
Alvin Bruney [MVP ASP.NET]

[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99


"Redhairs" wrote in message
news:#GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
> In a web farm environment, how to store the user uploading files for
> future access?
> Store them in db, local file system or centralized file server?
>
> If trying to storing the file in local file system or centralized file
> server, how the web server
> receives the uploading file and sync to each web server's local file
> system? or the centralized
> file server?
>
> Should I create only one folder or lots of subfolders to organize the file
> structure for better access
> from file system?
>

Re: File Storage Question

Thanks for your reply.
Just like simple photo album web app for user to upload image files and
review them later.


"Alvin Bruney [ASP.NET MVP]" wrote in message
news:789FFECD-922A-40BC-8B64-E2E618D45126@microsoft.com...
> Depends on what you want to do with the files and what they represent.
> Since you haven't said, ASP.NET uploads files into server memory, buffers
> it and then writes it to disk in the simplest case. Where you choose to
> store it depends on how frequently you want to access it and the
> responsiveness you want from the application, among other things.
>
> --
> --
> Regards,
> Alvin Bruney [MVP ASP.NET]
>
> [Shameless Author plug]
> The O.W.C. Black Book, 2nd Edition
> Exclusively on www.lulu.com/owc $19.99
>
>
> "Redhairs" wrote in message
> news:#GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
>> In a web farm environment, how to store the user uploading files for
>> future access?
>> Store them in db, local file system or centralized file server?
>>
>> If trying to storing the file in local file system or centralized file
>> server, how the web server
>> receives the uploading file and sync to each web server's local file
>> system? or the centralized
>> file server?
>>
>> Should I create only one folder or lots of subfolders to organize the
>> file structure for better access
>> from file system?
>>

Re: File Storage Question

In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.

In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.

In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...

--
Chris Mullins

"Redhairs" wrote in message
news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
> In a web farm environment, how to store the user uploading files for
> future access?
> Store them in db, local file system or centralized file server?
>
> If trying to storing the file in local file system or centralized file
> server, how the web server
> receives the uploading file and sync to each web server's local file
> system? or the centralized
> file server?
>
> Should I create only one folder or lots of subfolders to organize the file
> structure for better access
> from file system?
>

Re: File Storage Question

Chris,

What type of long term problems?
Can you share please I store images in a database.

Thank You,

LVP


"Chris Mullins [MVP - C#]" wrote in message
news:%23pCiTsmTIHA.1208@TK2MSFTNGP03.phx.gbl...
> In solutions that I've designed that have this as a requirement, I've
> typically used Network Attached Storage.
>
> In most cases, all the servers in the web farm are on the same gigabit
> network segment, and the NAS (which is usually multi-homed) sits on that
> same segment.
>
> In most cases, storing files in the database will lead to long term
> problems. As amazing as current databases are, they're still not file
> systems...
>
> --
> Chris Mullins
>
> "Redhairs" wrote in message
> news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
>> In a web farm environment, how to store the user uploading files for
>> future access?
>> Store them in db, local file system or centralized file server?
>>
>> If trying to storing the file in local file system or centralized file
>> server, how the web server
>> receives the uploading file and sync to each web server's local file
>> system? or the centralized
>> file server?
>>
>> Should I create only one folder or lots of subfolders to organize the
>> file structure for better access
>> from file system?
>>
>
>

Re: File Storage Question

Any tips on file structure design based on performance and maintenance
concerns?
e.g. one folder for all image files? or many subfolders?

"Chris Mullins [MVP - C#]" wrote in message
news:%23pCiTsmTIHA.1208@TK2MSFTNGP03.phx.gbl...
> In solutions that I've designed that have this as a requirement, I've
> typically used Network Attached Storage.
>
> In most cases, all the servers in the web farm are on the same gigabit
> network segment, and the NAS (which is usually multi-homed) sits on that
> same segment.
>
> In most cases, storing files in the database will lead to long term
> problems. As amazing as current databases are, they're still not file
> systems...
>
> --
> Chris Mullins
>
> "Redhairs" wrote in message
> news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
>> In a web farm environment, how to store the user uploading files for
>> future access?
>> Store them in db, local file system or centralized file server?
>>
>> If trying to storing the file in local file system or centralized file
>> server, how the web server
>> receives the uploading file and sync to each web server's local file
>> system? or the centralized
>> file server?
>>
>> Should I create only one folder or lots of subfolders to organize the
>> file structure for better access
>> from file system?
>>
>
>

Re: File Storage Question

The biggest problems I've seen with storing files in a database is that it
grows without bound, and current databases don't seem to do very well with
really large data sets. In the large range (1TB -> 1PB) the file systems (in
my experience) deals with large sets more effectivly. The database is great
for storing "Which computers have that image on it?", but for the actual
image.... not so hot.

If you are adding files quickly enough, you'll need to be adding disk space
into your database every hours / days / weeks. Even with current large
drives, this is both expensive and time consuming. Using a DB, there's
really no good way to scatter the data across a large number of computers -
you could use FileGroup in SQL2K5 (Enterprise only), or something similar,
but these are not very good for dynamically adding data.

... if you're not adding drives, you're archiving files. This means a job
that runs, and moves images from the DB to a disk. You then need to design
the system to first check the DB, then check the disk. This makes things 2x
as complicated as just skipping the DB step and storing files on disk.

My preference at this point is:
- Store files on a NAS, or a set of NAS devices.
- Store the index (file names, metadata, location[s]) in the database.

I would love to hear of an out-of-the-box approach people have used for
this - especially in Oracle land (I'm familar with the SQL Server
landscape).

--
Chris Mullins

"LVP" wrote in message
news:eJkM2GnTIHA.4360@TK2MSFTNGP06.phx.gbl...
> Chris,
>
> What type of long term problems?
> Can you share please I store images in a database.
>
> Thank You,
>
> LVP
>
>
> "Chris Mullins [MVP - C#]" wrote in message
> news:%23pCiTsmTIHA.1208@TK2MSFTNGP03.phx.gbl...
>> In solutions that I've designed that have this as a requirement, I've
>> typically used Network Attached Storage.
>>
>> In most cases, all the servers in the web farm are on the same gigabit
>> network segment, and the NAS (which is usually multi-homed) sits on that
>> same segment.
>>
>> In most cases, storing files in the database will lead to long term
>> problems. As amazing as current databases are, they're still not file
>> systems...
>>
>> --
>> Chris Mullins
>>
>> "Redhairs" wrote in message
>> news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
>>> In a web farm environment, how to store the user uploading files for
>>> future access?
>>> Store them in db, local file system or centralized file server?
>>>
>>> If trying to storing the file in local file system or centralized file
>>> server, how the web server
>>> receives the uploading file and sync to each web server's local file
>>> system? or the centralized
>>> file server?
>>>
>>> Should I create only one folder or lots of subfolders to organize the
>>> file structure for better access
>>> from file system?
>>>
>>
>>
>
>

Re: File Storage Question

This seems to be somewhat one sided argument, storing in the DB works well in
some situations, it can work well if the file sizes aren't too large. If you
are storing several TB of images then you probably don't want to put them in
the database, but if the sizing is smaller it should not be a problem.

The argument you make about disk space really isn't a database issue it is a
disk issue and you'll run out of disk space just as quickly in the file
system as in the DB.

The advantages to storing in the database are a central location and
syncronized backups between the images and the database. The primary problem
is really if you have a lot of clients(in this case web servers in the farm)
hitting the database they will be chewing up bandwidth into your db server to
get the images. Another disadvantage is as noted in parent post the file
system generally is faster.

However I have used SQL Server before for storing images for web use, in
these cases though I was limiting the actual file size of the image to 55KB
or less and made use of some caching on the webserver, I never saw a
performance problem. Also because I controlled the point of entry for the
images I would scale the images down with GDI+ server side before saving to
the db.

"Chris Mullins [MVP - C#]" wrote:

> The biggest problems I've seen with storing files in a database is that it
> grows without bound, and current databases don't seem to do very well with
> really large data sets. In the large range (1TB -> 1PB) the file systems (in
> my experience) deals with large sets more effectivly. The database is great
> for storing "Which computers have that image on it?", but for the actual
> image.... not so hot.
>
> If you are adding files quickly enough, you'll need to be adding disk space
> into your database every hours / days / weeks. Even with current large
> drives, this is both expensive and time consuming. Using a DB, there's
> really no good way to scatter the data across a large number of computers -
> you could use FileGroup in SQL2K5 (Enterprise only), or something similar,
> but these are not very good for dynamically adding data.
>
> ... if you're not adding drives, you're archiving files. This means a job
> that runs, and moves images from the DB to a disk. You then need to design
> the system to first check the DB, then check the disk. This makes things 2x
> as complicated as just skipping the DB step and storing files on disk.
>
> My preference at this point is:
> - Store files on a NAS, or a set of NAS devices.
> - Store the index (file names, metadata, location[s]) in the database.
>
> I would love to hear of an out-of-the-box approach people have used for
> this - especially in Oracle land (I'm familar with the SQL Server
> landscape).
>
> --
> Chris Mullins
>
> "LVP" wrote in message
> news:eJkM2GnTIHA.4360@TK2MSFTNGP06.phx.gbl...
> > Chris,
> >
> > What type of long term problems?
> > Can you share please I store images in a database.
> >
> > Thank You,
> >
> > LVP
> >
> >
> > "Chris Mullins [MVP - C#]" wrote in message
> > news:%23pCiTsmTIHA.1208@TK2MSFTNGP03.phx.gbl...
> >> In solutions that I've designed that have this as a requirement, I've
> >> typically used Network Attached Storage.
> >>
> >> In most cases, all the servers in the web farm are on the same gigabit
> >> network segment, and the NAS (which is usually multi-homed) sits on that
> >> same segment.
> >>
> >> In most cases, storing files in the database will lead to long term
> >> problems. As amazing as current databases are, they're still not file
> >> systems...
> >>
> >> --
> >> Chris Mullins
> >>
> >> "Redhairs" wrote in message
> >> news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
> >>> In a web farm environment, how to store the user uploading files for
> >>> future access?
> >>> Store them in db, local file system or centralized file server?
> >>>
> >>> If trying to storing the file in local file system or centralized file
> >>> server, how the web server
> >>> receives the uploading file and sync to each web server's local file
> >>> system? or the centralized
> >>> file server?
> >>>
> >>> Should I create only one folder or lots of subfolders to organize the
> >>> file structure for better access
> >>> from file system?
> >>>
> >>
> >>
> >
> >
>
>
>

Re: File Storage Question

On Jan 4, 8:45=A0pm, Jeff wrote:
> This seems to be somewhat one sided argument, storing in the DB works well=
in
> some situations, it can work well if the file sizes aren't too large. =A0I=
f you
> are storing several TB of images then you probably don't want to put them =
in
> the database, but if the sizing is smaller it should not be a problem. =A0=

>
> The argument you make about disk space really isn't a database issue it is=
a
> disk issue and you'll run out of disk space just as quickly in the file
> system as in the DB. =A0
>
> The advantages to storing in the database are a central location and
> syncronized backups between the images and the database. =A0The primary pr=
oblem
> is really if you have a lot of clients(in this case web servers in the far=
m)
> hitting the database they will be chewing up bandwidth into your db server=
to
> get the images. =A0Another disadvantage is as noted in parent post the fil=
e
> system generally is faster.
>
> However I have used SQL Server before for storing images for web use, in
> these cases though I was limiting the actual file size of the image to 55K=
B
> or less and made use of some caching on the webserver, I never saw a
> performance problem. =A0Also because I controlled the point of entry for t=
he
> images I would scale the images down with GDI+ server side before saving t=
o
> the db.
>
> "Chris Mullins [MVP - C#]" wrote:
>
>
>
> > The biggest problems I've seen with storing files in a database is that =
it
> > grows without bound, and current databases don't seem to do very well wi=
th
> > really large data sets. In the large range (1TB -> 1PB) the file systems=
(in
> > my experience) deals with large sets more effectivly. The database is gr=
eat
> > for storing "Which computers have that image on it?", but for the actual=

> > image.... not so hot.
>
> > If you are adding files quickly enough, you'll need to be adding disk sp=
ace
> > into your database every hours / days / weeks. Even with current large
> > drives, this is both expensive and time consuming. Using a DB, there's
> > really no good way to scatter the data across a large number of computer=
s -
> > you could use FileGroup in SQL2K5 (Enterprise only), or something simila=
r,
> > but these are not very good for dynamically adding data.
>
> > ... if you're not adding drives, you're archiving files. This means a jo=
b
> > that runs, and moves images from the DB to a disk. You then need to desi=
gn
> > the system to first check the DB, then check the disk. This makes things=
2x
> > as complicated as just skipping the DB step and storing files on disk.
>
> > My preference at this point is:
> > - Store files on a NAS, or a set of NAS devices.
> > - Store the index (file names, metadata, location[s]) in the database.
>
> > I would love to hear of an out-of-the-box approach people have used for
> > this - especially in Oracle land (I'm familar with the SQL Server
> > landscape).
>
> > --
> > Chris Mullins
>
> > "LVP" wrote in message
> >news:eJkM2GnTIHA.4360@TK2MSFTNGP06.phx.gbl...
> > > Chris,
>
> > > What type of long term problems?
> > > Can you share please I store images in a database.
>
> > > Thank You,
>
> > > LVP
>
> > > "Chris Mullins [MVP - C#]" wrote in message
> > >news:%23pCiTsmTIHA.1208@TK2MSFTNGP03.phx.gbl...
> > >> In solutions that I've designed that have this as a requirement, I've=

> > >> typically used Network Attached Storage.
>
> > >> In most cases, all the servers in the web farm are on the same gigabi=
t
> > >> network segment, and the NAS (which is usually multi-homed) sits on t=
hat
> > >> same segment.
>
> > >> In most cases, storing files in the database will lead to long term
> > >> problems. As amazing as current databases are, they're still not file=

> > >> systems...
>
> > >> --
> > >> Chris Mullins
>
> > >> "Redhairs" wrote in message
> > >>news:%23GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
> > >>> In a web farm environment, how to store the user uploading files for=

> > >>> future access?
> > >>> Store them in db, local file system or centralized file server?
>
> > >>> If trying to storing the file in local file system or centralized fi=
le
> > >>> server, how the web server
> > >>> receives the uploading file and sync to each web server's local file=

> > >>> system? or the centralized
> > >>> file server?
>
> > >>> Should I create only one folder or lots of subfolders to organize th=
e
> > >>> file structure for better access
> > >>> from file system?- Hide quoted text -
>
> - Show quoted text -

To BLOB or Not To BLOB: Large Object Storage in a Database or a
Filesystem?
http://research.microsoft.com/research/pubs/view.aspx?msr_tr _id=3DMSR-TR-200=
6-45

Re: File Storage Question

"Alexey Smirnov" wrote
> To BLOB or Not To BLOB: Large Object Storage in
> a Database or a Filesystem?
> http://research.microsoft.com/research/pubs/view.aspx?msr_tr _id=MSR-TR-2006-45

That's really a great paper.

I quite enjoyed reading it - having real metrics applied to this stuff is
exactly what I like to see!

--
Chris Mullins

Re: File Storage Question

You may want to consider a tree like hierarchy where the top folder
represents the logged on user account with a sub folder that has the image
files. It's easier to trouble shoot or retrieve images that way since you
index on the logged on user and then read all the files in that folder. When
you get the file upload from the user, simply find the logged on user
account and store the file inside the directory. If this is anonymous, that
approach won't work.

--
--
Regards,
Alvin Bruney [MVP ASP.NET]

[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99


"Redhairs" wrote in message
news:egzMCAmTIHA.5980@TK2MSFTNGP04.phx.gbl...
> Thanks for your reply.
> Just like simple photo album web app for user to upload image files and
> review them later.
>
>
> "Alvin Bruney [ASP.NET MVP]" wrote in message
> news:789FFECD-922A-40BC-8B64-E2E618D45126@microsoft.com...
>> Depends on what you want to do with the files and what they represent.
>> Since you haven't said, ASP.NET uploads files into server memory, buffers
>> it and then writes it to disk in the simplest case. Where you choose to
>> store it depends on how frequently you want to access it and the
>> responsiveness you want from the application, among other things.
>>
>> --
>> --
>> Regards,
>> Alvin Bruney [MVP ASP.NET]
>>
>> [Shameless Author plug]
>> The O.W.C. Black Book, 2nd Edition
>> Exclusively on www.lulu.com/owc $19.99
>>
>>
>> "Redhairs" wrote in message
>> news:#GHoU4jTIHA.5788@TK2MSFTNGP06.phx.gbl...
>>> In a web farm environment, how to store the user uploading files for
>>> future access?
>>> Store them in db, local file system or centralized file server?
>>>
>>> If trying to storing the file in local file system or centralized file
>>> server, how the web server
>>> receives the uploading file and sync to each web server's local file
>>> system? or the centralized
>>> file server?
>>>
>>> Should I create only one folder or lots of subfolders to organize the
>>> file structure for better access
>>> from file system?
>>>
>
>