High GET- retrieval failure rate

Hello,

I am using libwww-perl 5.80 with perl 5.80 and get a high GET-
retrieval failure rate (70-80%) when I connect to a particular server
located in Israel (I am based in Europe).

When I use curl instead of LWP the failure rate is almost 0%.

Has anyone had this problem before? Do I need to upgrade both Perl and
LWP?

Philippe

--------- My script ----------------

#!/usr/bin/perl -w
use strict;

use LWP::UserAgent;

if ( ! @ARGV ) {
die "Usage: $0 ";
}

my $number_connections = $ARGV[0];
my $timeout = $ARGV[1];
my $get_this = 'http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';

my $dateext = `date +%s`;
chomp $dateext;
my $file = "lwptest$dateext.log";
my $count_successes = 0;
my $count_failures = 0;

print "Connecting to $get_this.\n";
print FH "Connecting to $get_this.\n";
print "Number of connections is $number_connections. Time-out is
$timeout second(s).\n";
print FH "Number of connections is $number_connections. Time-out is
$timeout second(s).\n";

for (my $i = 0; $i < $number_connections; $i++) {
my $ua = LWP::UserAgent->new;
$ua->agent("INA picker v0.1");
$ua->timeout($timeout);
my $req = HTTP::Request->new(GET => $get_this);
#$req->authorization_basic($partner_retrieve_pointer->{'user name'},
# $partner_retrieve_pointer->{'password'});
my $res = $ua->request($req);
my $date = `date +'%H:%M:%S'`;
chomp $date;
printf( "Connection %d/%d (%3.2f%%) ", ($i+1), $number_connections,
((($i+1)/$number_connections)*100) );
if ($res->is_success) {
print "succeeded.\n";
$count_successes++;
} else {
print "failed. Reason: " . $res->status_line . "\n";
$count_failures++;
}
}

my $result = (($count_failures/$number_connections)*100);
printf("Failure rate: %3.2f%%.\n", $result);

Re: High GET- retrieval failure rate

--------------060107010209020109010009
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hello Philippe,

I have been using Mechanize, rather than LWP directly, so I am not sure
I can be of much help. When I was debugging my connections, one useful
tip I got was to turn on LWP's debugging. Include this line in your script:

use LWP::Debug qw(+);

This way you can see what is happening at the protocol level.

I run queries on several different servers on a daily basis. My problems
all appear to be time of day issues ( = scheduled maintenance). When
logging into some sites, I have had some problems with the site not
being ready for the followup request until after a few seconds have gone
by, but this is a server problem, not a connectivity issue, and was
solved by sleeping for a few seconds.

The actual URL would be helpful to reproce the problem.

Good Luck!

Peter

Philippe de Rochambeau wrote:

> Hello,
>
> I am using libwww-perl 5.80 with perl 5.80 and get a high GET-
> retrieval failure rate (70-80%) when I connect to a particular server
> located in Israel (I am based in Europe).
>
> When I use curl instead of LWP the failure rate is almost 0%.
>
> Has anyone had this problem before? Do I need to upgrade both Perl and
> LWP?
>
> Philippe
>
> --------- My script ----------------
>
> #!/usr/bin/perl -w
> use strict;
>
> use LWP::UserAgent;
>
> if ( ! @ARGV ) {
> die "Usage: $0 ";
> }
>
> my $number_connections = $ARGV[0];
> my $timeout = $ARGV[1];
> my $get_this = 'http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
>
> my $dateext = `date +%s`;
> chomp $dateext;
> my $file = "lwptest$dateext.log";
> my $count_successes = 0;
> my $count_failures = 0;
>
> print "Connecting to $get_this.\n";
> print FH "Connecting to $get_this.\n";
> print "Number of connections is $number_connections. Time-out is
> $timeout second(s).\n";
> print FH "Number of connections is $number_connections. Time-out is
> $timeout second(s).\n";
>
> for (my $i = 0; $i < $number_connections; $i++) {
> my $ua = LWP::UserAgent->new;
> $ua->agent("INA picker v0.1");
> $ua->timeout($timeout);
> my $req = HTTP::Request->new(GET => $get_this);
> #$req->authorization_basic($partner_retrieve_pointer->{'user name'},
> # $partner_retrieve_pointer->{'password'});
> my $res = $ua->request($req);
> my $date = `date +'%H:%M:%S'`;
> chomp $date;
> printf( "Connection %d/%d (%3.2f%%) ", ($i+1),
> $number_connections, ((($i+1)/$number_connections)*100) );
> if ($res->is_success) {
> print "succeeded.\n";
> $count_successes++;
> } else {
> print "failed. Reason: " . $res->status_line . "\n";
> $count_failures++;
> }
> }
>
> my $result = (($count_failures/$number_connections)*100);
> printf("Failure rate: %3.2f%%.\n", $result);
>
>

--------------060107010209020109010009--