How can I "PUT" a large file?

Hi all,

I need to perform a PUT operation and send a very large file (several
hundred MBytes). I have been using the following code to do this:

...
my $header = HTTP::Headers->new;
$header->content_type('application/octet-stream');
$header->content_length($fileSize);
$header->authorization_basic($usr, $pwd);

my $readFunc = sub {
read(FH, my $buf, 65536);
return $buf;
};

my $req = HTTP::Request->new("PUT", $url, $header, $readFunc);
...

But after updating to the 5.802 version of LWP this code has stopped
working.

When I execute my script, it prints a warning telling me that the
Content-Length header has been fixed, and the file in the destination
server is corrupted.
Looking at the code of the library, I have found these lines:

# Set (or override) Content-Length header
my $clen = $request_headers->header('Content-Length');
if (defined($$content_ref) && length($$content_ref)) {
$has_content++;
if (!defined($clen) || $clen ne length($$content_ref)) {
if (defined $clen) {
warn "Content-Length header value was wrong, fixed";
hlist_remove(\@h, 'Content-Length');
}
push(@h, 'Content-Length' => length($$content_ref));
}
} elsif ($clen) {
warn "Content-Length set when there is not content, fixed";
hlist_remove(\@h, 'Content-Length');
}

I think these lines prevent the use of a function as the content reference.

Is this a bug, or the support for function references has been removed?

Thanks in advance,
Rodrigo Ruiz


--
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.813 / Virus Database: 553 - Release Date: 13/12/2004

Re: How can I "PUT" a large file?

Rodrigo Ruiz writes:

> I need to perform a PUT operation and send a very large file (several
> hundred MBytes). I have been using the following code to do this:
>
> ...
> my $header = HTTP::Headers->new;
> $header->content_type('application/octet-stream');
> $header->content_length($fileSize);
> $header->authorization_basic($usr, $pwd);
>
> my $readFunc = sub {
> read(FH, my $buf, 65536);
> return $buf;
> };
>
> my $req = HTTP::Request->new("PUT", $url, $header, $readFunc);
> ...

Seems sane.

> But after updating to the 5.802 version of LWP this code has stopped
> working.
>
> When I execute my script, it prints a warning telling me that the
> Content-Length header has been fixed, and the file in the destination
> server is corrupted.
> Looking at the code of the library, I have found these lines:
>
> # Set (or override) Content-Length header
> my $clen = $request_headers->header('Content-Length');
> if (defined($$content_ref) && length($$content_ref)) {
> $has_content++;
> if (!defined($clen) || $clen ne length($$content_ref)) {
> if (defined $clen) {
> warn "Content-Length header value was wrong, fixed";
> hlist_remove(\@h, 'Content-Length');
> }
> push(@h, 'Content-Length' => length($$content_ref));
> }
> } elsif ($clen) {
> warn "Content-Length set when there is not content, fixed";
> hlist_remove(\@h, 'Content-Length');
> }
>
> I think these lines prevent the use of a function as the content reference.
>
> Is this a bug, or the support for function references has been removed?

No this is supposed to work. This code block should not be entered as
there is a test for code reference content just above it. Can you
figure out why the:

if (ref($content_ref) eq 'CODE') {

test fails? What is $content_ref in this case?

Regards,
Gisle

Re: How can I "PUT" a large file?

Gisle Aas writes:

> No this is supposed to work.

I've now verified that using request code content like this, does
indeed work for me when posting to my own server.

Unless you can debug this problem directly with your app, please try
to create a complete (short) example program the demonstrates this
problem and send it to this list.

Regards,
Gisle

Re: How can I "PUT" a large file?

Sorry, my apologies by my mistake :_(

At last, I was having a library path confusion.

The code works fine, but I was editing the wrong source file.

The file that was being executed had this code:

my $ref = ($LWP::VERSION >= 5.8) ? \$readFunc : $readFunc;
my $req = HTTP::Request->new("PUT", $url, $header, $ref);

To fix a problem that appeared in version 5.8 of the library.

I was editing an identical file, but at a different location, so my
changes were not taking effect at all :-P
Once I have found the correct file, and replaced the ">=" by an "==",
all started to work again.

The error appeared on 5.8 version of LWP. My current version is 5.802,
and it has the error fixed. Is this the first version where the bug is
fixed? Is it enough to do an "==" comparison or should I use something like:

my $ref = ($LWP::VERSION >= 5.8 && $LWP::VERSION < 5.802) ? \$readFunc :
$readFunc;


Thanks for your help, once I knew the code should have worked it took me
just a few seconds to guess where the real problem was :-D




--
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.813 / Virus Database: 553 - Release Date: 13/12/2004

Re: How can I "PUT" a large file?

Rodrigo Ruiz writes:

> The error appeared on 5.8 version of LWP. My current version is 5.802,
> and it has the error fixed. Is this the first version where the bug is
> fixed? Is it enough to do an "==" comparison or should I use something
> like:
>
> my $ref = ($LWP::VERSION >= 5.8 && $LWP::VERSION < 5.802) ? \$readFunc
> : $readFunc;

This bug was only present in one version; libwww-perl-5.800. If you
really still need this workaround I would make it:

my $readFunc = sub { .... };
$readFunc = \$readFunc if $LWP::VERSION eq "5.800"; # workaround buggy LWP version

Regards,
Gisle

Re: How can I "PUT" a large file?

I will follow your advice :-)

Thank you very much,
Rodrigo

> This bug was only present in one version; libwww-perl-5.800. If you
really still need this workaround I would make it:
>
> my $readFunc = sub { .... };
> $readFunc = \$readFunc if $LWP::VERSION eq "5.800"; # workaround
buggy LWP version
>
> Regards,
> Gisle


--
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.813 / Virus Database: 553 - Release Date: 14/12/2004