LWP::Protocol::http bug with SSL returning 500 Can"t read entity body

Hello,

I'm using libwww-perl-5.805.tar.gz.

Running code like this to get a page:

use WWW::Mechanize;
our $Mech = WWW::Mechanize->new();
#$Mech->get("https://library.minlib.net:443/patroninfo/");

Returns a "500 Can't read entity body: " error.

This is because the read_entity_body loop can't cope with a multi sysread
where the last packet returned is the EOF alone.

It's the first time I've poked in this code, so this may be off, but one
possible patch that fixes it is below. This is for

# $Id: http.pm,v 1.70 2005/12/08 10:28:01 gisle Exp $

Thanks for the great package.

*** /usr/lib/perl5/site_perl/5.8.6/LWP/Protocol/http.pm.old 2006-03-05 21:14:32.000000000 -0500
--- /usr/lib/perl5/site_perl/5.8.6/LWP/Protocol/http.pm 2006-03-05 21:04:35.000000000 -0500
***************
*** 338,349 ****
--- 338,352 ----
$response->push_header('Client-Response-Num', $socket->increment_response_count);

my $complete;
+ my $gotone;
$response = $self->collect($arg, $response, sub {
my $buf = ""; #prevent use of uninitialized value in SSLeay.xs
my $n;
READ:
{
$n = $socket->read_entity_body($buf, $size);
+ $gotone = 1 if defined $n;
+ $n=0 if !defined $n && $gotone;
die "Can't read entity body: $!" unless defined $n;
redo READ if $n == -1;
}

Re: LWP::Protocol::http bug with SSL returning 500 Can"t read entity body

"Wilson Snyder" writes:

> I'm using libwww-perl-5.805.tar.gz.
>
> Running code like this to get a page:
>
> use WWW::Mechanize;
> our $Mech = WWW::Mechanize->new();
> #$Mech->get("https://library.minlib.net:443/patroninfo/");
>
> Returns a "500 Can't read entity body: " error.

It doesn't fail when I tried it (even after uncommenting the get
line). I used Crypt::SSLeay; what SSL module did you use?

> This is because the read_entity_body loop can't cope with a multi sysread
> where the last packet returned is the EOF alone.

I don't understand this. What is the sequence of sysread returns in
this case? EOF is different from a failed read.

> It's the first time I've poked in this code, so this may be off, but one
> possible patch that fixes it is below.

The loop in LWP::Protocol::http is correct so I will not patch it.
The error is at a different level; either in read_entity_body() or
in the SSL module.

Regards,
Gisle

Re: LWP::Protocol::http bug with SSL returning 500 Can"t read entity body

>> I'm using libwww-perl-5.805.tar.gz.
>>
>> Running code like this to get a page:
>>
>> use WWW::Mechanize;
>> our $Mech = WWW::Mechanize->new();
>> #$Mech->get("https://library.minlib.net:443/patroninfo/");
>>
>> Returns a "500 Can't read entity body: " error.
>
>It doesn't fail when I tried it (even after uncommenting the get
>line). I used Crypt::SSLeay; what SSL module did you use?

Net::SSLeay 1.30.

>> This is because the read_entity_body loop can't cope with a multi sysread
>> where the last packet returned is the EOF alone.
>
>I don't understand this. What is the sequence of sysread returns in
>this case? EOF is different from a failed read.

sysread of EOF is supposed to return zero, but in other network programs
I've seen a IO::Socket->sysread return of undef treated as == a EOF. (Don't
ask me to prove the reasoning... :) It may be when the EOF was detected by
an earlier read, and then the next sysread comes and the socket is now
fully closed, so returns undef for reading the closed socket.

Anyhow, here's the calls:

calling read_entity_body(size=4096), returned 823
my_read calls sysread(buf, 4096) returns r=4096, $!=''
calling read_entity_body(size=4096), returned 4096
my_read calls sysread(buf, 4096) returns r=1388, $!=''
calling read_entity_body(size=4096), returned 1387
my_read calls sysread(buf, 4096) returns r=undef, $!=''
calling read_entity_body(size=4096), returned undef

The second sysread returns a buffer ending in '' and a newline. Looking at the page source
in firefox, that is the end of the page.

>> It's the first time I've poked in this code, so this may be off, but one
>> possible patch that fixes it is below.
>
>The loop in LWP::Protocol::http is correct so I will not patch it.
>The error is at a different level; either in read_entity_body() or
>in the SSL module.

Quite likely, as I stated I have little clue what the internals do, all I
know is it fixed - or at least hid - my case :)

Maybe read_entity_body should then return 0 in the last packet?

-Wilson