script to create to create tar and place files in the appropriate folder

--20cf305497e94d40a504a48c5471
Content-Type: text/plain; charset=ISO-8859-1

HI


i am given ever day a list of files which i wget within a parent folder .

my team gives me a structure where theses files are located

================
cat /home/upload_files1.txt
download the files form wget http://localserver/website1 and upload on
http://prodserver/weebsite1

index.php
dualfilder/dual.templ dual_2.templ
go/go.php
go/numbers/123.php
data/data.php
===============


using the above data given to me i check whether the files exist and if
they do i download them

wget http://localserver/website1/index.php
wget http://localserver/website1/dualfilder/dual.templ
wget http://localserver/website1/dualfilder/dual_2.templ
wget http://localserver/website1/go/go.php
wget http://localserver/website1/go/numbers/123.php
wget http://localserver/website1/data/data.php

and its downloaded in the /temp/ dir :

ls

index.php 123.php dual.templ dual_2.templ go.php data.txt

i need to create a tar of this in the same given format

index.php
dualfilder/dual.templ dual_2.templ
go/go.php
go/numbers/123.php
data/data.php


i cant seem to come up with a a logic on how to get this done .. can some
one help me kick start a idea on how to preceded .


Thanks a lot for all your help


--
Regards
Agnello D'souza

--20cf305497e94d40a504a48c5471--

Re: script to create to create tar and place files in the appropriate folder

--002215975fe211a72104a48de852
Content-Type: text/plain; charset=ISO-8859-1

On May 31, 2011 2:16 AM, "Agnello George" wrote:
>
> HI
>
>
> i am given ever day a list of files which i wget within a parent folder
..
>
> my team gives me a structure where theses files are located
>
> ================
> cat /home/upload_files1.txt
> download the files form wget http://localserver/website1 and upload on
> http://prodserver/weebsite1
>
> index.php
> dualfilder/dual.templ dual_2.templ
> go/go.php
> go/numbers/123.php
> data/data.php
> ===============
>

If this file is created by hand, you'll have some fun (sarc).

It looks like you're given a download and upload uri. Are you going to
assume they'll both always be present or not?

>
> using the above data given to me i check whether the files exist and if
> they do i download them
>
[Snip]
>
> i need to create a tar of this in the same given format
>
> index.php
> dualfilder/dual.templ dual_2.templ
> go/go.php
> go/numbers/123.php
> data/data.php
>
>
> i cant seem to come up with a a logic on how to get this done .. can some
> one help me kick start a idea on how to preceded .
>

I don't see much (complex) logic needed here. Just figure what format your
input should be, read in the variables, go.

To get the pages try LWP::Simple. I'm sure there's a module to tar for you.
However if there's not it shouldn't be hard for you to make a good enough
module of your own for tar.

--002215975fe211a72104a48de852--

Re: script to create to create tar and place files in the appropriatefolder

On 31/05/2011 07:15, Agnello George wrote:
> HI
>
>
> i am given ever day a list of files which i wget within a parent folder .
>
> my team gives me a structure where theses files are located
>
> ================
> cat /home/upload_files1.txt
> download the files form wget http://localserver/website1 and upload on
> http://prodserver/weebsite1
>
> index.php
> dualfilder/dual.templ dual_2.templ
> go/go.php
> go/numbers/123.php
> data/data.php
> ===============
>
>
> using the above data given to me i check whether the files exist and if
> they do i download them
>
> wget http://localserver/website1/index.php
> wget http://localserver/website1/dualfilder/dual.templ
> wget http://localserver/website1/dualfilder/dual_2.templ
> wget http://localserver/website1/go/go.php
> wget http://localserver/website1/go/numbers/123.php
> wget http://localserver/website1/data/data.php
>
> and its downloaded in the /temp/ dir :
>
> ls
>
> index.php 123.php dual.templ dual_2.templ go.php data.txt
>
> i need to create a tar of this in the same given format
>
> index.php
> dualfilder/dual.templ dual_2.templ
> go/go.php
> go/numbers/123.php
> data/data.php
>
>
> i cant seem to come up with a a logic on how to get this done .. can some
> one help me kick start a idea on how to preceded .

From what you have written, it sounds like you have already got the
download part in place. (If not, or you would like to take a walk
through your code, then please post again.)

You are best off using an established module to write the tar file
(indeed, unless you are already committed to it, you should use the Perl
LWP module over calls to wget) and Archive::Tar works just fine.

For instance:

use strict;
use warnings;

use Archive::Tar;

my @tarfiles;


# Download remote files and push them onto the @tarfiles array.

Archive::Tar->create_archive('tarfile.tar', undef, @tarfiles) or die
Archive::Tar->error;

__END__


HTH,

Rob

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriate folder

--000e0cd56a54c2e7bb04a4955b41
Content-Type: text/plain; charset=ISO-8859-1

> ls
>>
>> index.php 123.php dual.templ dual_2.templ go.php data.txt
>>
>> i need to create a tar of this in the same given format
>>
>> index.php
>> dualfilder/dual.templ dual_2.templ
>> go/go.php
>> go/numbers/123.php
>> data/data.php
>>
>>
>> i cant seem to come up with a a logic on how to get this done .. can some
>> one help me kick start a idea on how to preceded .
>>
>
> From what you have written, it sounds like you have already got the
> download part in place. (If not, or you would like to take a walk
> through your code, then please post again.)
>
> You are best off using an established module to write the tar file
> (indeed, unless you are already committed to it, you should use the Perl
> LWP module over calls to wget) and Archive::Tar works just fine.
>
> For instance:
>
> use strict;
> use warnings;
>
> use Archive::Tar;
>
> my @tarfiles;
>
>
> # Download remote files and push them onto the @tarfiles array.
>
> Archive::Tar->create_archive('tarfile.tar', undef, @tarfiles) or die
> Archive::Tar->error;
>
>

the difficult part is not taring the file but to recreate the folder
structure in the same format

when i wget all the files from the local server i get all of them under one
folder /temp

ls


index.php
dual.templ
dual_2.templ
go.php
123.php
data.php

but i need to tar it in such a way that the folder structure is preserved
like this

index.php
dualfilder/dual.templ
dualfilder/dual_2.templ
go/go.php
go/numbers/123.php
data/data.php









--
Regards
Agnello D'souza

--000e0cd56a54c2e7bb04a4955b41--

Re: script to create to create tar and place files in the appropriate folder

On Tue, May 31, 2011 at 2:15 AM, Agnello George
wrote:
> i am given ever day a list of files  which i wget  within a par=
ent folder .
>
> my  team gives me    a structure where theses files are lo=
cated
*snip*
> using the above data given to me  i check whether the files exist =
 and if
> they do i download them
*snip*
> i need to create a tar of this in the same given format



I like the idea of computers doing what people otherwise have to
manually do (often tediously). :) That's sort of what made me
want to be a programmer. :) While you hear other people complain
about how long certain data tasks require, you can often think of
a simple program to cut hours or days down into seconds or
minutes.

In any case, I wrote a program to do as much as I could make
sense of from the OP's requirements. I'm not entirely sure I
understood correctly, but either way I made some assumptions
about the format of the file he is given. :) I assumed the first
line would always contain the source server and the destination
server in that order preceded by the words "download" and
"upload" respectively. I assumed the next line would be blank and
every line after that would be a path/file part, optionally
followed by more file names (which should be joined with the
directory of the first file on the line). For example:

foo/bar baz

Would represent foo/bar and foo/baz.

Once I have that parsed I dump it to the user and prompt them to
continue (so that if the file is malformed or the parsing just
screws up it doesn't waste any time or do anything naughty). Once
the user says to continue, it uses LWP::Simple to fetch the
files, and Archive::Tar to put them into a tarball (preserving
the tree structure). Since I don't know what you do with the
tarball that's as far as I could get so the example program by
default dumps the Archive::Tar object, but if passed any
arguments, it writes the tarball to the first argument.

For example:

perl foo.pl blah.tar

Would write the tarball to blah.tar.

Currently the instructions are coming from DATA, but the function
that parses it accepts a file handle so you can instead open a
file handle to the file system and pass that (or modify it to
handle strings as file names or something). I did it for fun so
now you get to have fun criticizing it. ;D It hasn't been tested
thoroughly so there are probably a few bugs and there are
definitely going to be some areas that need improvement before
you could call it robust. I used my own Web server as the source
for this example so that I could actually test the LWP::Simple
and Archive::Tar stuff.

@OP: if you need the tarball compressed then I noticed that
Archive::Tar::write accepts extra arguments to specify that, but
I didn't bother for this example. :)

Note that the first line of the instructions won't fit on a
single line in an E-mail so it will definitely wrap. :) It is up
to you to fix that before running the program (though the error
handling should catch it if you don't).


#!/usr/bin/env perl

use strict;
use warnings;

use Data::Dumper;
use LWP::Simple;
use Archive::Tar;

main(@ARGV);

sub fetch_files
{
my ($src_uri, $file_names) =3D @_;

$src_uri =3D~ s{[\\/]$}{};

my %files;

for my $file (@{$file_names})
{
$files{$file} =3D get("$src_uri/$file") or
warn "Failed to get '$file'";
}

return \%files;
}

sub main
{
my @args =3D @_;
my $instructions =3D parse_instructions(\*DATA);
prompt_to_continue($instructions);
my $files =3D fetch_files(@$instructions{qw(src_uri files)});
my $tar =3D make_tarball($files);

# I don't know how you plan to upload it to the server so
# I'll just write it out to the file system instead. :)
$tar->write($ARGV[0]) if @args;
print Dumper $tar unless @args;
}

sub make_tarball
{
my ($files) =3D @_;

my $tar =3D Archive::Tar->new;

for my $file (keys %{$files})
{
$tar->add_data($file, $files->{$file});
}

return $tar;
}

sub parse_instructions
{
my ($fh) =3D @_;

my @lines =3D <$fh>;

die "Malformed instructions file" unless @lines >=3D 2;

my $uri_re =3D qr(http://[^ ]+);

my ($src_uri, $dest_uri) =3D
$lines[0] =3D~
m{download.*?($uri_re).*upload.*?($uri_re)};

die "Failed to parse server URIs"
unless defined $src_uri && defined $dest_uri;

my $start_line =3D 2;

unless($lines[1] =3D~ /^\s*$/)
{
warn "The second line isn't empty" ;
$start_line--;
}

@lines =3D @lines[$start_line..$#lines];

my @files;

for my $line (@lines)
{
my @dirfiles =3D split ' ', $line;

next unless @dirfiles;

@dirfiles =3D @dirfiles[1..$#dirfiles]
if $dirfiles[0] =3D~ /^\s+$/;

if(@dirfiles > 1)
{
my ($directory) =3D $dirfiles[0] =3D~ m{^(.*)/[^/]+$};

warn "Failed to parse directory"
unless defined $directory;

map { s{(.*)}{$directory/$1}; $_; }
@dirfiles[1..$#dirfiles];
}

push @files, @dirfiles;
}

return {
src_uri =3D> $src_uri,
dest_uri =3D> $dest_uri,
files =3D> \@files
};
}

sub prompt_to_continue
{
my ($instructions) =3D @_;
print Dumper $instructions;
print "Continue? (yes/no) ";
exit(1) unless =3D~ /^y(?:es)?$/i;
}

__DATA__
download the files form wget http://castopulence.org  and upload on
http://prodserver/weebsite1

news.html
projects.xhtml
downloads.xml
js/acc.dev.js acc.js
schitzwin/index.html schitzwin.pl



--=20
Brandon McCaig
V zrna gur orfg jvgu jung V fnl. Vg qbrfa'g nyjnlf fbhaq gung jnl.
Castopulence Software ..org>

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriatefolder

Brandon McCaig wrote:
>

Hello,

Just a couple of comments on some of your code. :)


>
> my $start_line = 2;
>
> unless($lines[1] =~ /^\s*$/)
> {
> warn "The second line isn't empty" ;
> $start_line--;
> }
>
> @lines = @lines[$start_line..$#lines];

You are copying almost all of @lines to @lines when you should be using
perl's built-in functions to just remove elements from @lines without
the copying:

splice @lines, 0, $start_line;



>
> map { s{(.*)}{$directory/$1}; $_; }
> @dirfiles[1..$#dirfiles];

What is $_; returning a value to? You are using map in a void context.

That is better written as:

s{(.*)}{$directory/$1} for @dirfiles[ 1 .. $#dirfiles ];

Or as:

$_ = "$directory/$_" for @dirfiles[ 1 .. $#dirfiles ];




John
--
Any intelligent fool can make things bigger and
more complex... It takes a touch of genius -
and a lot of courage to move in the opposite
direction. -- Albert Einstein

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriatefolder

>>>>> "JWK" == John W Krahn writes:

JWK> Or as:

JWK> $_ = "$directory/$_" for @dirfiles[ 1 .. $#dirfiles ];

just to show another way that is usually faster for prepending a string:

substr( $_, 0, 0, "$directory/" ) for @dirfiles[ 1 .. $#dirfiles ];

that is called 4 arg substr and it is way underused in perl. this
benchmark shows the significant speedup of about 2x:


use Benchmark qw( cmpthese ) ;

cmpthese ( shift || -2, {

assign => sub { my $x = 'abc' ; $x = "qwerty/$x" },
substr => sub { my $x = 'abc' ; substr( $x, 0, 0, 'qwerty/') },
}
) ;


Rate assign substr
assign 2356409/s -- -47%
substr 4428292/s 88% --






--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriate folder

On Tue, May 31, 2011 at 4:23 PM, John W. Krahn wrote:
> Just a couple of comments on some of your code.  :)
>
*snip*
>>     @lines =3D @lines[$start_line..$#lines];
>
> You are copying almost all of @lines to @lines when you should be using
> perl's built-in functions to just remove elements from @lines without the
> copying:
>
>       splice @lines, 0, $start_line;

I simply didn't know about splice and knew I could use a slice. :) I
certainly prefer the splice option though. Thank you for pointing it
out.

>>
>>             map { s{(.*)}{$directory/$1}; =
$_; }
>>                     @d=
irfiles[1..$#dirfiles];
>
> What is $_; returning a value to?  You are using map in a void conte=
xt.

Good catch. I'm still pretty green with the use of map or grep, etc.
(basically anything that accepts a BLOCK like that) so as you can see
I'm rather clumsy when I use them. :) Thanks for pointing this out
too. :)

> That is better written as:
>
>               s{(.*)}{$directory/$1} f=
or @dirfiles[ 1 .. $#dirfiles ];
>
> Or as:
>
>               $_ =3D "$directory/$_" f=
or @dirfiles[ 1 .. $#dirfiles ];

I can't believe I opted for a substitution instead of simply
"interpolating" the string... :-X I'll have to blame sed/vim for that
instinctual reaction. :P I guess the way that Perl aliases things
still confuses me a little bit. :)

Thanks again.


--=20
Brandon McCaig
V zrna gur orfg jvgu jung V fnl. Vg qbrfa'g nyjnlf fbhaq gung jnl.
Castopulence Software ..org>

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriate folder

On Tue, May 31, 2011 at 4:34 PM, Uri Guttman wrote:
> just to show another way that is usually faster for prepending a string:
>
>        substr( $_, 0, 0, "$directory/" ) for @dirfile=
s[ 1 .. $#dirfiles ];
>
> that is called 4 arg substr and it is way underused in perl. this
> benchmark shows the significant speedup of about 2x:

Thanks, Uri. :) I never would have thought substr would support such a
feature. In fact, I just read the perldoc and it's quite impressive
how it can work as an lvalue too. That's what I love about Perl. :)


--=20
Brandon McCaig
V zrna gur orfg jvgu jung V fnl. Vg qbrfa'g nyjnlf fbhaq gung jnl.
Castopulence Software ..org>

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriatefolder

>>>>> "BM" == Brandon McCaig writes:

BM> On Tue, May 31, 2011 at 4:34 PM, Uri Guttman wr=
ote:
>> just to show another way that is usually faster for prepending a strin=
g:
>>=20
>> =A0 =A0 =A0 =A0substr( $_, 0, 0, "$directory/" ) for @dirfiles[ 1 .. $=
#dirfiles ];
>>=20
>> that is called 4 arg substr and it is way underused in perl. this
>> benchmark shows the significant speedup of about 2x:

BM> Thanks, Uri. :) I never would have thought substr would support such a
BM> feature. In fact, I just read the perldoc and it's quite impressive
BM> how it can work as an lvalue too. That's what I love about Perl. :)

here's the catch. lvalue substr is older but actually very slow due to
how it is implemented. 4 arg substr is much faster and does the same
thing so use that and avoid lvalue.

what lvalue does is mark a section of a string which you can assign
into. 4 arg can do the same thing by marking the same string part and
instead of an assignment it just takes the inserted string as an arg. so
you see lvalue has to create something which can then be used by the
assignment op which is somewhat complicated. 4 arg can just take the
string and insert it where the substr tells it to.

there are some obscure useful cases for lvalue substr but for all normal
cases of inserting a string (and optionally replacing a part) use 4 arg
substr.

uri

--=20
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com =
--
----- Perl Code Review , Architecture, Development, Training, Support ----=
--
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com -------=
--

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriatefolder

Uri Guttman wrote:
>>>>>> "JWK" == John W Krahn writes:
>
> JWK> Or as:
>
> JWK> $_ = "$directory/$_" for @dirfiles[ 1 .. $#dirfiles ];
>
> just to show another way that is usually faster for prepending a string:
>
> substr( $_, 0, 0, "$directory/" ) for @dirfiles[ 1 .. $#dirfiles ];
>
> that is called 4 arg substr and it is way underused in perl. this
> benchmark shows the significant speedup of about 2x:
>
>
> use Benchmark qw( cmpthese ) ;
>
> cmpthese ( shift || -2, {
>
> assign => sub { my $x = 'abc' ; $x = "qwerty/$x" },
> substr => sub { my $x = 'abc' ; substr( $x, 0, 0, 'qwerty/') },
> }
> ) ;
>
>
> Rate assign substr
> assign 2356409/s -- -47%
> substr 4428292/s 88% --

I'm not seeing it:

$ perl -le'
use Benchmark qw( cmpthese );
cmpthese ( shift || -2, {
assign => sub { my $x = "abc"; my $dir = "qwerty"; $x = "$dir/$x" },
substr => sub { my $x = "abc"; my $dir = "qwerty"; substr $x, 0, 0,
"$dir/" },
} );
'
Rate substr assign
substr 997968/s -- -2%
assign 1018371/s 2% --




John
--
Any intelligent fool can make things bigger and
more complex... It takes a touch of genius -
and a lot of courage to move in the opposite
direction. -- Albert Einstein

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriatefolder

>>>>> "JWK" == John W Krahn writes:

JWK> Uri Guttman wrote:
>>>>>>> "JWK" == John W Krahn writes:
>>
>> that is called 4 arg substr and it is way underused in perl. this
>> benchmark shows the significant speedup of about 2x:
>>
>>
>> use Benchmark qw( cmpthese ) ;
>>
>> cmpthese ( shift || -2, {
>>
>> assign => sub { my $x = 'abc' ; $x = "qwerty/$x" },
>> substr => sub { my $x = 'abc' ; substr( $x, 0, 0, 'qwerty/') },
>> }
>> ) ;
>>
>>
>> Rate assign substr
>> assign 2356409/s -- -47%
>> substr 4428292/s 88% --

JWK> I'm not seeing it:

JWK> $ perl -le'
JWK> use Benchmark qw( cmpthese );
JWK> cmpthese ( shift || -2, {
JWK> assign => sub { my $x = "abc"; my $dir = "qwerty"; $x = "$dir/$x" },
JWK> substr => sub { my $x = "abc"; my $dir = "qwerty"; substr $x, 0,
JWK> 0, "$dir/" },
JWK> } );
JWK> '

JWK> Rate substr assign
JWK> substr 997968/s -- -2%
JWK> assign 1018371/s 2% --

you are doing more work than my version. so i benchmarked all 4
variations and this is the result.


cmpthese ( shift || -2, {

assign => sub { my $x = 'abc' ; $x = "qwerty/$x" },
substr => sub { my $x = 'abc' ; substr( $x, 0, 0, 'qwerty/') },
assign2 => sub { my $x = 'abc' ; my $z = 'qwerty' ;
$x = "$z/$x" },
substr2 => sub { my $x = 'abc' ; my $z = 'qwerty' ;
substr( $x, 0, 0, "$z/" ) },
}
) ;

Rate assign2 substr2 assign substr
assign2 2383123/s -- -1% -13% -39%
substr2 2407127/s 1% -- -12% -38%
assign 2739050/s 15% 14% -- -30%
substr 3896373/s 63% 62% 42% --

what i see is that the extra assignment to $z is making that the largest
part of the cpu load and hides the difference of the substr vs
assignment. one key to benchmarking is to isolate the actual code you
want to compare. in this case there are two plain assignments which
overwhelm the difference on the code under comparison. my original
benchmark (show here again) highlights the difference better. if i could
get rid of the needed initialization assignment i would.

note that this benchmark will also be affected by the length of the
strings involved (both the initialized and the prepended). you can
likely design a set of strings that will make either look faster than
the other.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: script to create to create tar and place files in the appropriate folder

--000e0cd56a542d184d04a4a059f1
Content-Type: text/plain; charset=ISO-8859-1



> push @files, @dirfiles;
> }
>
> return {
> src_uri => $src_uri,
> dest_uri => $dest_uri,
> files => \@files
> };
> }
>
> sub prompt_to_continue
> {
> my ($instructions) = @_;
> print Dumper $instructions;
> print "Continue? (yes/no) ";
> exit(1) unless =~ /^y(?:es)?$/i;
> }
>
> __DATA__
> download the files form wget http://castopulence.org and upload on
> http://prodserver/weebsite1
>
> news.html
> projects.xhtml
> downloads.xml
> js/acc.dev.js acc.js
> schitzwin/index.html schitzwin.pl
>
>


wow !! you have actually come up with with a solution ... i kind of new to
perl .. and will take some time to understand and read you code . .. thanks
a lot .


--
Regards
Agnello D'souza

--000e0cd56a542d184d04a4a059f1--