Provide a link to huge files | ftp to windows

I am developing some shell scripts which eventually produce huge data
files - they are about 2-10 GB huge. The business requirement is to
place these files in two or three different paths on the same server.
Our knee jerk reaction to this requirement was to just copy the files
over to the other paths and leave them there.

But the problem is that these files are so huge that they are
constantly causing diskspace issues. So I was wondering if there was a
way I could place the file in one central place and provide a link to
this file from all the other different paths.

Is that achievable?

Also - these files get ftp-ed to windows boxes using an ftp client -
would the links (if do-able) cause a problem to the ftp process?

We use the Hummingbird Exceed ftp client - if that makes a difference.

Thanks,
Anoop

Re: Provide a link to huge files | ftp to windows

Anoop wrote:
> I am developing some shell scripts which eventually produce huge data
> files - they are about 2-10 GB huge. The business requirement is to
> place these files in two or three different paths on the same server.
> Our knee jerk reaction to this requirement was to just copy the files
> over to the other paths and leave them there.
>
> But the problem is that these files are so huge that they are
> constantly causing diskspace issues. So I was wondering if there was a
> way I could place the file in one central place and provide a link to
> this file from all the other different paths.
>
> Is that achievable?
>
> Also - these files get ftp-ed to windows boxes using an ftp client -
> would the links (if do-able) cause a problem to the ftp process?
>
> We use the Hummingbird Exceed ftp client - if that makes a difference.
>
> Thanks,
> Anoop
>


create soft link and do this kind of things.

Re: Provide a link to huge files | ftp to windows

Hey Anoop,

Vakyail is correct (though not giving any explanation), and there is
very little reason for
having a 20 gig file copied six places on the same machine -- so long
multiple people are
not needing to make changes to that file, and/or have separate/
different versions.

Look at the "ln" command, and it's 'man page'.

Effectively, you can have ONE file, with MANY file names pointing to
it under unix.
When V. is saying create a link -- he means "make multiple names for
that same file"

a "hard link" actually makes many file names for the same data
(potentially in different dirs)
a "soft link" makes a pointer to the original file, almost like a web-
link that explains where the file is stored.
Note: hard links must be on the same file-system/mount-point/logical-
disk as the stored data file.

So, your main file may exist under one directory, like /var/reports/
weekly/my_big_file.dat
Your can create soft links to the file like this:

ln -s /var/reports/weekly/my_big_file.dat /usr/ref/my_big_file.dat
ln -s /var/reports/weekly/my_big_file.dat /public/info/
my_big_file.dat
ln -s /var/reports/weekly/my_big_file.dat /home/shared/
big_file_pointer.dat
ln -s /var/reports/weekly/my_big_file.dat /opt/another/listing/
Nov_Fat_Report.txt

When you look at the Link files, you will see a directory listing
like:

lrwxr--r-- 1 bob dev 34 /usr/ref/my_big_file.dat -> /
var/reports/weekly/my_big_file.dat

which shows you that this is not the actual file, but just a pointer
to the file itself.

When you delete the file, you will also need to remember to clean-up
the links too, unless you
are just going to put a new version of the file down in the same place
& same name.

That should be enough to get you moving.

Re: Provide a link to huge files | ftp to windows

On Nov 1, 12:26 am, Jstein wrote:
> Hey Anoop,
>
> Vakyail is correct (though not giving any explanation), and there is
> very little reason for
> having a 20 gig file copied six places on the same machine -- so long
> multiple people are
> not needing to make changes to that file, and/or have separate/
> different versions.
>
> Look at the "ln" command, and it's 'man page'.
>
> Effectively, you can have ONE file, with MANY file names pointing to
> it under unix.
> When V. is saying create a link -- he means "make multiple names for
> that same file"
>
> a "hard link" actually makes many file names for the same data
> (potentially in different dirs)
> a "soft link" makes a pointer to the original file, almost like a web-
> link that explains where the file is stored.
> Note: hard links must be on the same file-system/mount-point/logical-
> disk as the stored data file.
>
> So, your main file may exist under one directory, like /var/reports/
> weekly/my_big_file.dat
> Your can create soft links to the file like this:
>
> ln -s /var/reports/weekly/my_big_file.dat /usr/ref/my_big_file.dat
> ln -s /var/reports/weekly/my_big_file.dat /public/info/
> my_big_file.dat
> ln -s /var/reports/weekly/my_big_file.dat /home/shared/
> big_file_pointer.dat
> ln -s /var/reports/weekly/my_big_file.dat /opt/another/listing/
> Nov_Fat_Report.txt
>
> When you look at the Link files, you will see a directory listing
> like:
>
> lrwxr--r-- 1 bob dev 34 /usr/ref/my_big_file.dat -> /
> var/reports/weekly/my_big_file.dat
>
> which shows you that this is not the actual file, but just a pointer
> to the file itself.
>
> When you delete the file, you will also need to remember to clean-up
> the links too, unless you
> are just going to put a new version of the file down in the same place
> & same name.
>
> That should be enough to get you moving.

That perfectly answers my question - thanks to both of you.

But if a windows user is ftp-ing the files, he should not see anything
different right? He will just see a file and when he "gets" the s-link
file, he will get the actual file ftp-ed over?

Thank you so much.

Re: Provide a link to huge files | ftp to windows

In article <1193927046.999753.159290@57g2000hsv.googlegroups.com>,
Anoop wrote:

> On Nov 1, 12:26 am, Jstein wrote:
> > Hey Anoop,
> >
> > Vakyail is correct (though not giving any explanation), and there is
> > very little reason for
> > having a 20 gig file copied six places on the same machine -- so long
> > multiple people are
> > not needing to make changes to that file, and/or have separate/
> > different versions.
> >
> > Look at the "ln" command, and it's 'man page'.
> >
> > Effectively, you can have ONE file, with MANY file names pointing to
> > it under unix.
> > When V. is saying create a link -- he means "make multiple names for
> > that same file"
> >
> > a "hard link" actually makes many file names for the same data
> > (potentially in different dirs)
> > a "soft link" makes a pointer to the original file, almost like a web-
> > link that explains where the file is stored.
> > Note: hard links must be on the same file-system/mount-point/logical-
> > disk as the stored data file.
> >
> > So, your main file may exist under one directory, like /var/reports/
> > weekly/my_big_file.dat
> > Your can create soft links to the file like this:
> >
> > ln -s /var/reports/weekly/my_big_file.dat /usr/ref/my_big_file.dat
> > ln -s /var/reports/weekly/my_big_file.dat /public/info/
> > my_big_file.dat
> > ln -s /var/reports/weekly/my_big_file.dat /home/shared/
> > big_file_pointer.dat
> > ln -s /var/reports/weekly/my_big_file.dat /opt/another/listing/
> > Nov_Fat_Report.txt
> >
> > When you look at the Link files, you will see a directory listing
> > like:
> >
> > lrwxr--r-- 1 bob dev 34 /usr/ref/my_big_file.dat -> /
> > var/reports/weekly/my_big_file.dat
> >
> > which shows you that this is not the actual file, but just a pointer
> > to the file itself.
> >
> > When you delete the file, you will also need to remember to clean-up
> > the links too, unless you
> > are just going to put a new version of the file down in the same place
> > & same name.
> >
> > That should be enough to get you moving.
>
> That perfectly answers my question - thanks to both of you.
>
> But if a windows user is ftp-ing the files, he should not see anything
> different right? He will just see a file and when he "gets" the s-link
> file, he will get the actual file ftp-ed over?

Correct. Almost all operations on a soft link access the actual file,
not the link itself. The exceptions are mostly commands that are
specifically intended to deal with file names rather than file contents
(e.g. ls, mv).

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Re: Provide a link to huge files | ftp to windows

On Nov 1, 11:43 am, Barry Margolin wrote:
> In article <1193927046.999753.159...@57g2000hsv.googlegroups.com>,
>
>
>
> Anoop wrote:
> > On Nov 1, 12:26 am, Jstein wrote:
> > > Hey Anoop,
>
> > > Vakyail is correct (though not giving any explanation), and there is
> > > very little reason for
> > > having a 20 gig file copied six places on the same machine -- so long
> > > multiple people are
> > > not needing to make changes to that file, and/or have separate/
> > > different versions.
>
> > > Look at the "ln" command, and it's 'man page'.
>
> > > Effectively, you can have ONE file, with MANY file names pointing to
> > > it under unix.
> > > When V. is saying create a link -- he means "make multiple names for
> > > that same file"
>
> > > a "hard link" actually makes many file names for the same data
> > > (potentially in different dirs)
> > > a "soft link" makes a pointer to the original file, almost like a web-
> > > link that explains where the file is stored.
> > > Note: hard links must be on the same file-system/mount-point/logical-
> > > disk as the stored data file.
>
> > > So, your main file may exist under one directory, like /var/reports/
> > > weekly/my_big_file.dat
> > > Your can create soft links to the file like this:
>
> > > ln -s /var/reports/weekly/my_big_file.dat /usr/ref/my_big_file.dat
> > > ln -s /var/reports/weekly/my_big_file.dat /public/info/
> > > my_big_file.dat
> > > ln -s /var/reports/weekly/my_big_file.dat /home/shared/
> > > big_file_pointer.dat
> > > ln -s /var/reports/weekly/my_big_file.dat /opt/another/listing/
> > > Nov_Fat_Report.txt
>
> > > When you look at the Link files, you will see a directory listing
> > > like:
>
> > > lrwxr--r-- 1 bob dev 34 /usr/ref/my_big_file.dat -> /
> > > var/reports/weekly/my_big_file.dat
>
> > > which shows you that this is not the actual file, but just a pointer
> > > to the file itself.
>
> > > When you delete the file, you will also need to remember to clean-up
> > > the links too, unless you
> > > are just going to put a new version of the file down in the same place
> > > & same name.
>
> > > That should be enough to get you moving.
>
> > That perfectly answers my question - thanks to both of you.
>
> > But if a windows user is ftp-ing the files, he should not see anything
> > different right? He will just see a file and when he "gets" the s-link
> > file, he will get the actual file ftp-ed over?
>
> Correct. Almost all operations on a soft link access the actual file,
> not the link itself. The exceptions are mostly commands that are
> specifically intended to deal with file names rather than file contents
> (e.g. ls, mv).
>
> --
> Barry Margolin, bar...@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***

Fantastic - thanks to all.