chunked transfer encoding and HTTP::Daemon

Does anybody have (or can point me to) a working example of
HTTP::Daemon that implements a server that is capable to handle
"Transfer-Encoding: chunked"? (I mean really capable, which means it
should work without a Content-Length header and for inputs that tickle
in in small chunks.)

I find the manpage a hard nut to crack with regard to cases that try
to handle the input themselves. And I have the impression that for
chunked transfer encoding I do have to handle the input myself.

Other solutions in perl welcome. Damn, even working solutions in other
environments welcome:-)

Thanks,
--
andreas

Re: chunked transfer encoding and HTTP::Daemon

--=-=-=

>>>>> On 07 Dec 2005 10:17:36 -0800, Gisle Aas said:

> andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:
>> Does anybody have (or can point me to) a working example of
>> HTTP::Daemon that implements a server that is capable to handle
>> "Transfer-Encoding: chunked"? (I mean really capable, which means it
>> should work without a Content-Length header and for inputs that tickle
>> in in small chunks.)

> The $c->get_request method should handle "Transfer-Encoding: chunked"
> just fine, but it will not return until all the content data has been
> received. If you want to process chunked data as it is received, then
> you will have to tell it to only read past the headers with
> $c->get_request(1) and then do the parsing of the chunked body
> yourself.

Thanks for coming back to my question. In the meantime I have verified
that $c->get_request() does it right. I append a testscript for that;
hope you like it.

What I had problems to get to work was the $c->get_request(1) + "do
the parsing yourself" variant. It would be a great help if there were
a sample implementation (maybe it would fit into my testscript?) that
is more or less bullet proof.

--
andreas

--=-=-=--

Re: chunked transfer encoding and HTTP::Daemon

andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:

> >>>>> On 07 Dec 2005 10:17:36 -0800, Gisle Aas said:
>
> > andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:
> >> Does anybody have (or can point me to) a working example of
> >> HTTP::Daemon that implements a server that is capable to handle
> >> "Transfer-Encoding: chunked"? (I mean really capable, which means it
> >> should work without a Content-Length header and for inputs that tickle
> >> in in small chunks.)
>
> > The $c->get_request method should handle "Transfer-Encoding: chunked"
> > just fine, but it will not return until all the content data has been
> > received. If you want to process chunked data as it is received, then
> > you will have to tell it to only read past the headers with
> > $c->get_request(1) and then do the parsing of the chunked body
> > yourself.
>
> Thanks for coming back to my question. In the meantime I have verified
> that $c->get_request() does it right. I append a testscript for that;
> hope you like it.

I did't find anything appended.

> What I had problems to get to work was the $c->get_request(1) + "do
> the parsing yourself" variant. It would be a great help if there were
> a sample implementation (maybe it would fit into my testscript?) that
> is more or less bullet proof.

Inside the get_request method you will find a block that goes like this:

if ($te && lc($te) eq 'chunked') {
# Handle chunked transfer encoding
my $body = "";
CHUNK:
while (1) {
print STDERR "Chunked\n" if $DEBUG;
if ($buf =~ s/^([^\012]*)\012//) {
my $chunk_head = $1;
unless ($chunk_head =~ /^([0-9A-Fa-f]+)/) {
$self->send_error(400);
$self->reason("Bad chunk header $chunk_head");
return;
}
my $size = hex($1);
last CHUNK if $size == 0;

my $missing = $size - length($buf) + 2; # 2=CRLF at chunk end
# must read until we have a complete chunk
while ($missing > 0) {
print STDERR "Need $missing more bytes\n" if $DEBUG;
my $n = $self->_need_more($buf, $timeout, $fdset);
return unless $n;
$missing -= $n;
}
$body .= substr($buf, 0, $size);
substr($buf, 0, $size+2) = '';

}
else {
# need more data in order to have a complete chunk header
return unless $self->_need_more($buf, $timeout, $fdset);
}
}
$r->content($body);

# pretend it was a normal entity body
$r->remove_header('Transfer-Encoding');
$r->header('Content-Length', length($body));

my($key, $val);
FOOTER:
while (1) {
if ($buf !~ /\012/) {
# need at least one line to look at
return unless $self->_need_more($buf, $timeout, $fdset);
}
else {
$buf =~ s/^([^\012]*)\012//;
$_ = $1;
s/\015$//;
if (/^([\w\-]+)\s*:\s*(.*)/) {
$r->push_header($key, $val) if $key;
($key, $val) = ($1, $2);
}
elsif (/^\s+(.*)/) {
$val .= " $1";
}
elsif (!length) {
last FOOTER;
}
else {
$self->reason("Bad footer syntax");
return;
}
}
}
$r->push_header($key, $val) if $key;

}

This is basically what you need to do :)

--Gisle

Re: chunked transfer encoding and HTTP::Daemon

--=-=-=

>>>>> On 12 Dec 2005 10:00:04 -0800, Gisle Aas said:

> andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:
>> Thanks for coming back to my question. In the meantime I have verified
>> that $c->get_request() does it right. I append a testscript for that;
>> hope you like it.

> I did't find anything appended.

Ooops, sorry, I'll retry now.

>> What I had problems to get to work was the $c->get_request(1) + "do
>> the parsing yourself" variant. It would be a great help if there were
>> a sample implementation (maybe it would fit into my testscript?) that
>> is more or less bullet proof.

> Inside the get_request method you will find a block that goes like this:

> if ($te && lc($te) eq 'chunked') {
> # Handle chunked transfer encoding
> my $body = "";
> CHUNK:
> while (1) {
> print STDERR "Chunked\n" if $DEBUG;
> if ($buf =~ s/^([^\012]*)\012//) {
> my $chunk_head = $1;
> unless ($chunk_head =~ /^([0-9A-Fa-f]+)/) {
> $self->send_error(400);
> $self->reason("Bad chunk header $chunk_head");
> return;
> }
> my $size = hex($1);
> last CHUNK if $size == 0;

> my $missing = $size - length($buf) + 2; # 2=CRLF at chunk end
> # must read until we have a complete chunk
> while ($missing > 0) {
> print STDERR "Need $missing more bytes\n" if $DEBUG;
> my $n = $self->_need_more($buf, $timeout, $fdset);

Watch that the arguments to _need_more() are not supplied within this
snippet. I'd have to write a variation of the whole 210 lines of
get_request including the use of a seemingly private method (as its
name starts with an underscore) and dereferencing ${*$self}. You will
understand that this is a bit against the usual rules?

> [...SNIP...]

> This is basically what you need to do :)

Sure:) But maybe you have a slightly more ecouraging idea for a interface?

--
andreas


--=-=-=
Content-Disposition: inline; filename=http-daemon.t
Content-Transfer-Encoding: quoted-printable

#!/usr/bin/perl

use strict;
use warnings;

use HTTP::Daemon;
use Test::More;
use Time::HiRes qw(sleep);
our $CRLF;
use Socket qw($CRLF);

our $LOGGING =3D 0;

our @TESTS =3D (
{
expect =3D> 629,
comment =3D> "traditional, unchunked POST request",
raw =3D> "POST /cgi-bin/redir-TE.pl HTTP/1.1
User-Agent: UNTRUSTED/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 629
Host: localhost

JSR-205=3D0;font_small=3D15;png=3D1;jpg=3D1;alpha_channel=3D 256;JSR-82=3D0;=
JSR-135=3D1;mot-wt=3D0;JSR-75-pim=3D0;pointer_motion_event=3 D0;camera=3D1;f=
ree_memory=3D455472;heap_size=3D524284;cldc=3DCLDC-1.1;canva s_size_y=3D176;=
canvas_size_x=3D176;double_buffered=3D1;color=3D65536;JSR-12 0=3D1;JSR-184=
=3D1;JSR-180=3D0;JSR-75-file=3D0;push_socket=3D0;pointer_eve nt=3D0;nokia-ui=
=3D1;java_platform=3Dxxxxxxxxxxxxxxxxx/xxxxxxx;gif=3D1;midp= 3DMIDP-1.0 MIDP=
-2.0;font_large=3D22;sie-col-game=3D0;JSR-179=3D0;push_sms=3 D1;JSR-172=3D0;=
font_medium=3D18;fullscreen_canvas_size_y=3D220;fullscreen_c anvas_size_x=3D=
176;java_locale=3Dde;video_encoding=3Dencoding=3DJPEG&width= 3D176&height=3D=
182encoding=3DJPEG&width=3D176&height=3D220;"
},
{
expect =3D> 8,
comment =3D> "chunked with illegal Content-Length header; ti=
ny message",
raw =3D> "POST /cgi-bin/redir-TE.pl HTTP/1.1
Host: localhost
Content-Type: application/x-www-form-urlencoded
Content-Length: 8
Transfer-Encoding: chunked

8
icm.x=3Du2
0

",
},
{
expect =3D> 868,
comment =3D> "chunked with illegal Content-Length header; me=
dium sized",
raw =3D> "POST /cgi-bin/redir-TE.pl HTTP/1.1
Host:dev05
Connection:close
Content-Type:application/x-www-form-urlencoded
Content-Length:868
transfer-encoding:chunked

364
JSR-205=3D0;font_small=3D20;png=3D1;jpg=3D1;JSR-82=3D0;JSR-1 35=3D1;mot-wt=
=3D0;JSR-75-pim=3D0;http=3D1;pointer_motion_event=3D0;browse r_launch=3D1;fr=
ee_memory=3D733456;user_agent=3Dxxxxxxxxx/xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxx;heap_size=3D815080;cldc=3DCLDC-1.0; canvas_size_y=
=3D182;canvas_size_x=3D176;double_buffered=3D1;NAVIGATION PRESS=3D20;JSR-18=
4=3D0;JSR-120=3D1;color=3D32768;JSR-180=3D0;JSR-75-file=3D0; RIGHT SOFT KEY=
=3D22;NAVIGATION RIGHT=3D5;KEY *=3D42;push_socket=3D0;pointer_event=3D0;KEY=
#=3D35;KEY NUM 9=3D57;nokia-ui=3D0;KEY NUM 8=3D56;KEY NUM 7=3D55;KEY NUM 6=
=3D54;KEY NUM 5=3D53;gif=3D1;KEY NUM 4=3D52;NAVIGATION UP=3D1;KEY NUM 3=3D5=
1;KEY NUM 2=3D50;KEY NUM 1=3D49;midp=3DMIDP-2.0 VSCL-1.1.0;font_large=3D20;=
KEY NUM 0=3D48;sie-col-game=3D0;JSR-179=3D0;push_sms=3D1;JSR-172=3D0 ;NAVIGA=
TION LEFT=3D2;LEFT SOFT KEY=3D21;font_medium=3D20;fullscreen_canvas_size_y=
=3D204;fullscreen_canvas_size_x=3D176;https=3D1;NAVIGATION DOWN=3D6;java_lo=
cale=3Den-DE;
0

",
},
{
expect =3D> 1104,
comment =3D> "chunked correctly, size ~1k; base for the big =
next test",
raw =3D> "POST /cgi-bin/redir-TE.pl HTTP/1.1
User-Agent: UNTRUSTED/1.0
Content-Type: application/x-www-form-urlencoded
Host: localhost:80
Transfer-Encoding: chunked

450
JSR-205=3D0;font_small=3D15;png=3D1;jpg=3D1;jsr184_dithering =3D0;CLEAR/DELE=
TE=3D-8;JSR-82=3D0;alpha_channel=3D32;JSR-135=3D1;mot-wt=3D0 ;JSR-75-pim=3D0=
;http=3D1;pointer_motion_event=3D0;browser_launch=3D1;BACK/R ETURN=3D-11;cam=
era=3D1;free_memory=3D456248;user_agent=3Dxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;heap_size=3 D524284;cldc=3D=
CLDC-1.1;canvas_size_y=3D176;canvas_size_x=3D176;double_buff ered=3D1;NAVIGA=
TION PRESS=3D-5;JSR-184=3D1;JSR-120=3D1;color=3D65536;JSR-180=3D0 ;JSR-75-fi=
le=3D0;RIGHT SOFT KEY=3D-7;NAVIGATION RIGHT=3D-4;KEY *=3D42;push_socket=3D0=
;pointer_event=3D0;KEY #=3D35;KEY NUM 9=3D57;nokia-ui=3D1;KEY NUM 8=3D56;KE=
Y NUM 7=3D55;KEY NUM 6=3D54;KEY NUM 5=3D53;java_platform=3Dxxxxxxxxxxxxxxxx=
x/xxxxxxx;KEY NUM 4=3D52;gif=3D1;KEY NUM 3=3D51;NAVIGATION UP=3D-1;KEY NUM =
2=3D50;KEY NUM 1=3D49;midp=3DMIDP-1.0 MIDP-2.0;font_large=3D22;KEY NUM 0=3D=
48;sie-col-game=3D0;JSR-179=3D0;push_sms=3D1;JSR-172=3D0;NAV IGATION LEFT=3D=
-3;LEFT SOFT KEY=3D-6;jsr184_antialiasing=3D0;font_medium=3D18;fullscreen _c=
anvas_size_y=3D220;fullscreen_canvas_size_x=3D176;https=3D1; NAVIGATION DOWN=
=3D-2;java_locale=3Dde;video_encoding=3Dencoding=3DJPEG&widt h=3D176&height=
=3D182encoding=3DJPEG&width=3D176&height=3D220;
0

"
},
{
expect =3D> 1104*1024,
comment =3D> "chunked with many chunks",
raw =3D> ("POST /cgi-bin/redir-TE.pl HTTP/1.1
User-Agent: UNTRUSTED/1.0
Content-Type: application/x-www-form-urlencoded
Host: localhost:80
Transfer-Encoding: chunked

".("450
JSR-205=3D0;font_small=3D15;png=3D1;jpg=3D1;jsr184_dithering =3D0;CLEAR/DELE=
TE=3D-8;JSR-82=3D0;alpha_channel=3D32;JSR-135=3D1;mot-wt=3D0 ;JSR-75-pim=3D0=
;http=3D1;pointer_motion_event=3D0;browser_launch=3D1;BACK/R ETURN=3D-11;cam=
era=3D1;free_memory=3D456248;user_agent=3Dxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;heap_size=3 D524284;cldc=3D=
CLDC-1.1;canvas_size_y=3D176;canvas_size_x=3D176;double_buff ered=3D1;NAVIGA=
TION PRESS=3D-5;JSR-184=3D1;JSR-120=3D1;color=3D65536;JSR-180=3D0 ;JSR-75-fi=
le=3D0;RIGHT SOFT KEY=3D-7;NAVIGATION RIGHT=3D-4;KEY *=3D42;push_socket=3D0=
;pointer_event=3D0;KEY #=3D35;KEY NUM 9=3D57;nokia-ui=3D1;KEY NUM 8=3D56;KE=
Y NUM 7=3D55;KEY NUM 6=3D54;KEY NUM 5=3D53;java_platform=3Dxxxxxxxxxxxxxxxx=
x/xxxxxxx;KEY NUM 4=3D52;gif=3D1;KEY NUM 3=3D51;NAVIGATION UP=3D-1;KEY NUM =
2=3D50;KEY NUM 1=3D49;midp=3DMIDP-1.0 MIDP-2.0;font_large=3D22;KEY NUM 0=3D=
48;sie-col-game=3D0;JSR-179=3D0;push_sms=3D1;JSR-172=3D0;NAV IGATION LEFT=3D=
-3;LEFT SOFT KEY=3D-6;jsr184_antialiasing=3D0;font_medium=3D18;fullscreen _c=
anvas_size_y=3D220;fullscreen_canvas_size_x=3D176;https=3D1; NAVIGATION DOWN=
=3D-2;java_locale=3Dde;video_encoding=3Dencoding=3DJPEG&widt h=3D176&height=
=3D182encoding=3DJPEG&width=3D176&height=3D220;
"x1024)."0

")
},
);


my $tests =3D @TESTS;
plan tests =3D> $tests;

sub mywarn ($) {
return unless $LOGGING;
my($mess) =3D @_;
open my $fh, ">>", "http-daemon.out"
or die $!;
my $ts =3D localtime;
print $fh "$ts: $mess\n";
close $fh or die $!;
}


my $pid;
if ($pid =3D fork) {
sleep 4;
for my $t (0..$#TESTS) {
my $test =3D $TESTS[$t];
my $raw =3D $test->{raw};
$raw =3D~ s/\r?\n/$CRLF/mg;
if (0) {
open my $fh, "| socket localhost 8333" or die;
print $fh $test;
}
use IO::Socket::INET;
my $sock =3D IO::Socket::INET->new(
PeerAddr =3D> "127.0.0.1",
PeerPort =3D> 8333,
) or die;
if (0) {
for my $pos (0..length($raw)-1) {
print $sock substr($raw,$pos,1);
sleep 0.001;
}
} else {
print $sock $raw;
}
local $/;
my $resp =3D <$sock>;
close $sock;
my($got) =3D $resp =3D~ /\r?\n\r?\n(\d+)/s;
is($got,
$test->{expect},
"[$test->{expect}] $test->{comment}",
);
}
wait;
} else {
die "cannot fork: $!" unless defined $pid;
my $d =3D HTTP::Daemon->new(
LocalAddr =3D> '0.0.0.0',
LocalPort =3D> 8333,
ReuseAddr =3D> 1,
) or die;
mywarn "Starting new daemon as '$$'";
my $i;
LISTEN: while (my $c =3D $d->accept) {
my $r =3D $c->get_request;
mywarn sprintf "headers[%s] content[%s]", $r->headers->as_string, $r->c=
ontent;
my $res =3D HTTP::Response->new(200,undef,undef,length($r->content).$CR=
LF);
$c->send_response($res);
$c->force_last_request; # we're just not mature enough
$c->close;
undef($c);
last if ++$i >=3D $tests;
}
}



# Local Variables:
# mode: cperl
# cperl-indent-level: 2
# End:

--=-=-=--

Re: chunked transfer encoding and HTTP::Daemon

andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:

> >>>>> On 12 Dec 2005 10:00:04 -0800, Gisle Aas said:
>
> > andreas.koenig.gmwojprw@franz.ak.mind.de (Andreas J. Koenig) writes:
> >> Thanks for coming back to my question. In the meantime I have verified
> >> that $c->get_request() does it right. I append a testscript for that;
> >> hope you like it.
>
> > I did't find anything appended.
>
> Ooops, sorry, I'll retry now.

Got it and checked it in as t/local/chunked.t. Thanks!

> > while ($missing > 0) {
> > print STDERR "Need $missing more bytes\n" if $DEBUG;
> > my $n = $self->_need_more($buf, $timeout, $fdset);
>
> Watch that the arguments to _need_more() are not supplied within this
> snippet. I'd have to write a variation of the whole 210 lines of
> get_request including the use of a seemingly private method (as its
> name starts with an underscore) and dereferencing ${*$self}. You will
> understand that this is a bit against the usual rules?

Yes. The quoted code was just to illustrate what needs to be done and
_need_more() is to be regarded as a template for $self->sysread($buf,
1024) or so.

> Sure:) But maybe you have a slightly more ecouraging idea for a interface?

Sorry, no and not enough time to develop it right now either.

--Gisle

Re: chunked transfer encoding and HTTP::Daemon

>>>>> On 13 Dec 2005 02:38:11 -0800, Gisle Aas said:

> Yes. The quoted code was just to illustrate what needs to be done and
> _need_more() is to be regarded as a template for $self->sysread($buf,
> 1024) or so.

Ahh, thanks, this will help me...

>> Sure:) But maybe you have a slightly more ecouraging idea for a interface?

> Sorry, no and not enough time to develop it right now either.

No problem, you have helped me enough now.

--
andreas