Searching for Robust Download Automation

Hi,

Looking for a (free or commercial)software solution to the following.
I need to automate the download of critical data from various sources
to various targets. File types vary. It should compare source and
target file sizes and do an md5sum to ensure success, and retry
download in cases of failure. It should have good logging.

I know I could write this myself in bash(or higher level scripting),
if I had to, but I don't particularly want to reinvent the wheel,
especially when my wheel will end up being squarish.

Have searched google but all I get for hits is peer-to-peer sharing
software.

Thanks

Re: Searching for Robust Download Automation

On 2008-04-07, Brendan wrote:
> Hi,
>
> Looking for a (free or commercial)software solution to the following.
> I need to automate the download of critical data from various sources
> to various targets. File types vary. It should compare source and
> target file sizes and do an md5sum to ensure success, and retry
> download in cases of failure. It should have good logging.
>
> I know I could write this myself in bash(or higher level scripting),
> if I had to, but I don't particularly want to reinvent the wheel,
> especially when my wheel will end up being squarish.
>
> Have searched google but all I get for hits is peer-to-peer sharing
> software.
>
> Thanks

rsync sounds like what you're looking for.


--
Christopher Mattern

NOTICE
Thank you for noticing this new notice
Your noticing it has been noted
And will be reported to the authorities

Re: Searching for Robust Download Automation

matternc@comcast.net writes:
>On 2008-04-07, Brendan wrote:
>> Hi,
>>
>> Looking for a (free or commercial)software solution to the following.
>> I need to automate the download of critical data from various sources
>> to various targets. File types vary. It should compare source and
>> target file sizes and do an md5sum to ensure success, and retry
>> download in cases of failure. It should have good logging.
>>
>> I know I could write this myself in bash(or higher level scripting),
>> if I had to, but I don't particularly want to reinvent the wheel,
>> especially when my wheel will end up being squarish.
>>
>> Have searched google but all I get for hits is peer-to-peer sharing
>> software.
>>
>> Thanks
>
>rsync sounds like what you're looking for.
>

Or, if you need to run a command upon successful transfer of
the file (or group of files), UUCP.

-Greg
--
::::::::::::: Greg Andrews ::::: gerg@panix.com :::::::::::::
I have a map of the United States that's actual size.
-- Steven Wright

Re: Searching for Robust Download Automation

Thanks for your advice. rysnc is not suitable since the data files
are in transit. They will be staged soon after downloading. I'll look
at UUCP, but I have resigned myself to learning enough python to make
my own utility.

Re: Searching for Robust Download Automation

On Apr 8, 9:24=A0am, Brendan wrote:
> Thanks for your advice. =A0rysnc is not suitable since the data files
> are in transit. They will be staged soon after downloading. I'll look
> at UUCP, but I have resigned myself to learning enough python to make
> my own utility.

UUCP? Was that a joke?

Re: Searching for Robust Download Automation

On Mon, 07 Apr 2008 06:05:21 -0700, Brendan wrote:

> Hi,
>
> Looking for a (free or commercial)software solution to the following. I
> need to automate the download of critical data from various sources to
> various targets. File types vary. It should compare source and target
> file sizes and do an md5sum to ensure success, and retry download in
> cases of failure. It should have good logging.
>
> I know I could write this myself in bash(or higher level scripting), if
> I had to, but I don't particularly want to reinvent the wheel,
> especially when my wheel will end up being squarish.
>
> Have searched google but all I get for hits is peer-to-peer sharing
> software.
>
> Thanks

Check out rsync -it'll do most of that.

Re: Searching for Robust Download Automation

Brendan writes:
>On Apr 8, 9:24 am, Brendan wrote:
>> Thanks for your advice.  rysnc is not suitable since the data files
>> are in transit. They will be staged soon after downloading. I'll look
>> at UUCP, but I have resigned myself to learning enough python to make
>> my own utility.
>
>UUCP? Was that a joke?
>

I was half joking, since the original poster didn't mention a need
to execute a command upon successful transfer.

However I was half serious. I can't count the number of times I've
seen people post questions in this group about scripting automatic
file transfers that will detect when the transfer is completed (not
just interrupted) and perform some action on the transferred file
that usually involves running a Unix script/command/program on it.

The usual replies seem to involve shell/perl scripts to glue a non-
retrying transport program (e.g. ftp/scp/rsync) together with an
ad hoc methods for detecting corrupted/incomplete files, executing
commands on the successful ones, logging any errors from executing
those commands, and cleaning out the transferred files. All of which
is reinventing the wheel that UUCP has had for 20+ years.

UUCP has done the things I list above for 20+ years, has had very
fast transfer protocols for 10+ years, and the config steps to run
it over ssh aren't harder than for other transports like rsync.

The two downsides to UUCP are its config files are more arcane, having
come out of the Bronze Age of Unix utility software development, and
there are fewer websites offering config tips.

However, for the average person asking for this kind of automatic
transfer system, those hurdles are much easier to handle than the large,
complex programming task required to implement the usual suggestions.


Don't get me wrong. I'm not saying UUCP is a be-all, end-all. I'm
just saying it's overlooked in some situations where it's the right
tool for the job. Then it's scoffed at, as you did.

-Greg

--
::::::::::::::: Greg Andrews :::::: gerg@panix.com :::::::::::::::
It is easier to write an incorrect program
than understand a correct one.

Re: Searching for Robust Download Automation

Was not scoffing, but when I saw how old it was I did think you were
either joking/trolling.

Re: Searching for Robust Download Automation

In article <8a7c5ec9-74d3-47f3-b6b8-efe503cdc9a1@n1g2000prb.googlegroups.com>,
Brendan wrote:
>Was not scoffing, but when I saw how old it was I did think you were
>either joking/trolling.

I think the point is that to some people, old is good, but to others,
just the opposite mentality entails.

Re: Searching for Robust Download Automation

On Wed, 9 Apr 2008 16:46:11 +0000 (UTC), gazelle@xmission.xmission.com
(Kenny McCormack) wrote:

>I think the point is that to some people, old is good, but to others,
>just the opposite mentality entails.

There's been nothing good since about 1990. And it will never get any
better than that.


--
Webmail for Dialup Users
http://www.isp2dial.com/freeaccounts.html

Re: Searching for Robust Download Automation

In article ,
www.isp2dial.com wrote:
>On Wed, 9 Apr 2008 16:46:11 +0000 (UTC), gazelle@xmission.xmission.com
>(Kenny McCormack) wrote:
>
>>I think the point is that to some people, old is good, but to others,
>>just the opposite mentality entails.
>
>There's been nothing good since about 1990. And it will never get any
>better than that.

ITA.

Re: Searching for Robust Download Automation

Post removed (X-No-Archive: yes)

Re: Searching for Robust Download Automation

On 2008-04-09, Kenny McCormack wrote:
> In article ,
> www.isp2dial.com wrote:
>>On Wed, 9 Apr 2008 16:46:11 +0000 (UTC), gazelle@xmission.xmission.com
>>(Kenny McCormack) wrote:
>>
>>>I think the point is that to some people, old is good, but to others,
>>>just the opposite mentality entails.
>>
>>There's been nothing good since about 1990. And it will never get any
>>better than that.
>
> ITA.
>
Um, ITA? Intercollegiate Tennis Association? Internation Trade Administrtion?
International Trombone Association? Information Technology Associates?

--
Christopher Mattern

NOTICE
Thank you for noticing this new notice
Your noticing it has been noted
And will be reported to the authorities

Re: Searching for Robust Download Automation

Chris Mattern wrote:

>> ITA.
>>
> Um, ITA? Intercollegiate Tennis Association? Internation Trade
> Administrtion?
> International Trombone Association? Information Technology Associates?

I think that stands for "I Totally Agree".

--
All the commands are tested with bash and GNU tools, so they may use
nonstandard features. I try to mention when something is nonstandard (if
I'm aware of that), but I may miss something. Corrections are welcome.

Re: Searching for Robust Download Automation

On Wed, 09 Apr 2008 14:03:53 -0500, Chris Mattern
wrote:

>>>There's been nothing good since about 1990. And it will never get any
>>>better than that.
>>
>> ITA.
>>
>Um, ITA? Intercollegiate Tennis Association? Internation Trade Administrtion?
>International Trombone Association? Information Technology Associates?

http://www.internetslang.com/


--
Webmail for Dialup Users
http://www.isp2dial.com/freeaccounts.html

Re: Searching for Robust Download Automation

Brendan wrote:
> I know I could write this myself in bash(or higher level scripting),
> if I had to, but I don't =A0particularly want to reinvent the wheel,
> especially when my wheel will end up being squarish.

No, no. Don't do that. I recall an effort 15(?) years ago or
so, to reimplement the UUCP toolsuite is [Bourne] shell. It
failed....

Since you're querying a shell group, here's a robust FTP
frontend that is scriptable:

"niftp.ksh":
http://web.unixlabplus.com:8080/unix-prog/niftp/niftp/

=3DBrian

Re: Searching for Robust Download Automation

On Apr 9, 1:40=A0am, g...@panix.com (Greg Andrews) wrote:
> >UUCP? Was that a joke?
>
> I was half joking, since the original poster didn't mention a need
> to execute a command upon successful transfer.
>
> However I was half serious. =A0I can't count the number of times I've
> seen people post questions in this group about scripting automatic
> file transfers that will detect when the transfer is completed (not
> just interrupted) and perform some action on the transferred file
> that usually involves running a Unix script/command/program on it.
>
> The usual replies seem to involve shell/perl scripts to glue a non-
> retrying transport program (e.g. ftp/scp/rsync) together with an
> ad hoc methods for detecting corrupted/incomplete files, executing
> commands on the successful ones, logging any errors from executing
> those commands, and cleaning out the transferred files. =A0All of which
> is reinventing the wheel that UUCP has had for 20+ years.
>
> UUCP has done the things I list above for 20+ years, has had very
> fast transfer protocols for 10+ years, and the config steps to run
> it over ssh aren't harder than for other transports like rsync.
>
> The two downsides to UUCP are its config files are more arcane, having
> come out of the Bronze Age of Unix utility software development, and
> there are fewer websites offering config tips.
>
> However, for the average person asking for this kind of automatic
> transfer system, those hurdles are much easier to handle than the large,
> complex programming task required to implement the usual suggestions.
>
> Don't get me wrong. =A0I'm not saying UUCP is a be-all, end-all. =A0I'm
> just saying it's overlooked in some situations where it's the right
> tool for the job. =A0Then it's scoffed at, as you did.
>

I can't use UUCP because some of the data source nodes are not under
my control.

Re: Searching for Robust Download Automation

On Apr 9, 8:34=A0pm, bsh wrote:
> Since you're querying a shell group, here's a robust FTP
> frontend that is scriptable:
>
> "niftp.ksh":http://web.unixlabplus.com:8080/unix-prog/niftp/ niftp/
>
> =3DBrian

Link is not working for me.

Re: Searching for Robust Download Automation

In article <54741d48-1bd9-495f-9939-7ba3f4a6d50e@h1g2000prh.googlegroups.com>,
Brendan wrote:
>On Apr 9, 8:34 pm, bsh wrote:
>> Since you're querying a shell group, here's a robust FTP
>> frontend that is scriptable:
>>
>> "niftp.ksh":http://web.unixlabplus.com:8080/unix-prog/niftp/ niftp/
>>
>> =Brian
>
>Link is not working for me.

Try

At least it seems like the right thing. It looks like the program
flourished about 20 years ago and was rewritten about 10 years ago.


carl

--
carl lowenstein marine physical lab, u.c. san diego
clowenstein@ucsd.edu