How does Transfer-Encoding: chunked work with CGI?

--0__=0ABBFCBDDFD8D5C18f9e8a93df938690918c0ABBFCBDDFD8D5C1
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable



Hi,
I haven't been able to find any documentation on how (or if) CGI handle=
s
Transfer-Encoding: chunked. It looks to me like Apache is eating the ch=
unk
size line, as the first byte of data I read from stdin is the data I'm
posting. I can clearly see the chunk size line preceding the data in a
packet capture of the http post.

Does Apache CGI support Transfer-Encoding: chunked? Is there documentat=
ion
of how it works?

Thanks,
Chris=

--0__=0ABBFCBDDFD8D5C18f9e8a93df938690918c0ABBFCBDDFD8D5C1
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline
Content-transfer-encoding: quoted-printable

Hi,

I haven't been able to find any documentation on how (or if) CGI handle=
s Transfer-Encoding: chunked. It looks to me like Apache is eating the =
chunk size line, as the first byte of data I read from stdin is the dat=
a I'm posting. I can clearly see the chunk size line preceding the data=
in a packet capture of the http post.



Does Apache CGI support Transfer-Encoding: chunked? Is there documentat=
ion of how it works?



Thanks,

Chris=

--0__=0ABBFCBDDFD8D5C18f9e8a93df938690918c0ABBFCBDDFD8D5C1--

Re: How does Transfer-Encoding: chunked work withCGI?

Chris Cross wrote:
> Hi,
> I haven't been able to find any documentation on how (or if) CGI handles
> Transfer-Encoding: chunked. It looks to me like Apache is eating the
> chunk size line, as the first byte of data I read from stdin is the data
> I'm posting. I can clearly see the chunk size line preceding the data in
> a packet capture of the http post.
>
> Does Apache CGI support Transfer-Encoding: chunked? Is there
> documentation of how it works?

This is not an Apache question. The CGI spec doesn't deal with
chunked data, so any HTTP/1.1 implementation of CGI (Apache included)
has to de-chunk any chunked input data on behalf of the script.

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: How does Transfer-Encoding: chunked work with CGI?

--0__=0ABBFCBDDFDEB9FD8f9e8a93df938690918c0ABBFCBDDFDEB9FD
Content-type: text/plain; charset=US-ASCII



Nick Kew wrote on 09/11/2009 09:55:25 AM:
> Chris Cross wrote:
> > Hi,
> > I haven't been able to find any documentation on how (or if) CGI
handles
> > Transfer-Encoding: chunked. It looks to me like Apache is eating the
> > chunk size line, as the first byte of data I read from stdin is the
data
> > I'm posting. I can clearly see the chunk size line preceding the data
in
> > a packet capture of the http post.
> >
> > Does Apache CGI support Transfer-Encoding: chunked? Is there
> > documentation of how it works?
>
> This is not an Apache question. The CGI spec doesn't deal with
> chunked data, so any HTTP/1.1 implementation of CGI (Apache included)
> has to de-chunk any chunked input data on behalf of the script.
>
> --
> Nick Kew

Thanks for the quick reply Nick. The CONTENT_LENGTH env variable is not
being set when I post with Transfer-Encoding: chunked. Does that mean that
Apache doesn't support de-chunking for CGI?

Chris
--0__=0ABBFCBDDFDEB9FD8f9e8a93df938690918c0ABBFCBDDFDEB9FD
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

Nick Kew <nick@webthing.com> wrote on 09/11/2009 09:55:25 AM:

> Chris Cross wrote:

> > Hi,

> > I haven't been able to find any documentation on how (or if) CGI handles

> > Transfer-Encoding: chunked. It looks to me like Apache is eating the

> > chunk size line, as the first byte of data I read from stdin is the data

> > I'm posting. I can clearly see the chunk size line preceding the data in

> > a packet capture of the http post.

> >

> > Does Apache CGI support Transfer-Encoding: chunked? Is there

> > documentation of how it works?

>

> This is not an Apache question.  The CGI spec doesn't deal with

> chunked data, so any HTTP/1.1 implementation of CGI (Apache included)

> has to de-chunk any chunked input data on behalf of the script.

>

> --

> Nick Kew



Thanks for the quick reply Nick. The CONTENT_LENGTH env variable is not being set when I post with Transfer-Encoding: chunked. Does that mean that Apache doesn't support de-chunking for CGI?



Chris
--0__=0ABBFCBDDFDEB9FD8f9e8a93df938690918c0ABBFCBDDFDEB9FD--

Re: How does Transfer-Encoding: chunked work withCGI?

Chris Cross wrote:

> Thanks for the quick reply Nick. The CONTENT_LENGTH env variable is not
> being set when I post with Transfer-Encoding: chunked. Does that mean
> that Apache doesn't support de-chunking for CGI?

Hmmm. That sounds buggish (dammit, it's a long time since I did
anything nontrivial with CGI).

What Apache version, and what CGI implementation?

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: How does Transfer-Encoding: chunked work with CGI?

--0__=0ABBFCBDDFC3F4618f9e8a93df938690918c0ABBFCBDDFC3F461
Content-type: text/plain; charset=US-ASCII



Nick Kew wrote on 09/11/2009 10:31:46 AM:
> Hmmm. That sounds buggish (dammit, it's a long time since I did
> anything nontrivial with CGI).
>
> What Apache version, and what CGI implementation?

From my CGI env:

SERVER_SOFTWARE=Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with
Suhosin-Patch mod_perl/2.0.4 Perl/v5.10.0
....
GATEWAY_INTERFACE=CGI/1.1
SERVER_PROTOCOL=HTTP/1.1
REQUEST_METHOD=PUT
QUERY_STRING=large-file.xml
REQUEST_URI=/cgi-bin/chunked-upload?large-file.xml
SCRIPT_NAME=/cgi-bin/chunked-upload


Chris
--0__=0ABBFCBDDFC3F4618f9e8a93df938690918c0ABBFCBDDFC3F461
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

Nick Kew <nick@webthing.com> wrote on 09/11/2009 10:31:46 AM:

> Hmmm.  That sounds buggish (dammit, it's a long time since I did

> anything nontrivial with CGI).

>

> What Apache version, and what CGI implementation?



From my CGI env:



SERVER_SOFTWARE=Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_perl/2.0.4 Perl/v5.10.0

...

GATEWAY_INTERFACE=CGI/1.1

SERVER_PROTOCOL=HTTP/1.1

REQUEST_METHOD=PUT

QUERY_STRING=large-file.xml

REQUEST_URI=/cgi-bin/chunked-upload?large-file.xml

SCRIPT_NAME=/cgi-bin/chunked-upload





Chris
--0__=0ABBFCBDDFC3F4618f9e8a93df938690918c0ABBFCBDDFC3F461--

Re: How does Transfer-Encoding: chunked work withCGI?

Chris Cross wrote:
> Nick Kew wrote on 09/11/2009 10:31:46 AM:
> > Hmmm. That sounds buggish (dammit, it's a long time since I did
> > anything nontrivial with CGI).
> >
> > What Apache version, and what CGI implementation?
>
> From my CGI env:

That's not actually what I asked (many CGI implementations would
give the same). But it's probably enough to go on: a recent-
enough apache release, probably prefork, and probably mod_cgi
(any Ubuntu folks here know whether it could be something
different without the server admin having made a conscious choice)?

Quick sanity checks:

(1) if your test script just reads to EOF, does it get the right
(de-chunked) inputs?
(2) does it fail with any reasonably-standard CGI library?

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: How does Transfer-Encoding: chunked work with CGI?

--0__=0ABBFCBDDFE3EDCD8f9e8a93df938690918c0ABBFCBDDFE3EDCD
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable



Nick Kew wrote on 09/11/2009 11:18:03 AM:
>
> Chris Cross wrote:
> > Nick Kew wrote on 09/11/2009 10:31:46 AM:
> > > Hmmm. That sounds buggish (dammit, it's a long time since I did=

> > > anything nontrivial with CGI).
> > >
> > > What Apache version, and what CGI implementation?
> >
> > From my CGI env:
>
> That's not actually what I asked (many CGI implementations would
> give the same). But it's probably enough to go on: a recent-
> enough apache release, probably prefork, and probably mod_cgi
> (any Ubuntu folks here know whether it could be something
> different without the server admin having made a conscious choice)?
>
> Quick sanity checks:
>
> (1) if your test script just reads to EOF, does it get the right
> (de-chunked) inputs?
> (2) does it fail with any reasonably-standard CGI library?
>
> --
> Nick Kew

Nick,

1) Reading till EOF works fine. I'd gotten in the habit of using
Content-Length in the non-chunked case so hadn't tried it until now. Th=
e
"Transfer-Encoding: chunked" header is set. Is that proper if "de-chunk=
ing"
has occurred?

2) I'm just doing C++ without any third party libraries.

So my test program is working now but I have another questions. Can you=

elaborate on what is a CGI implementation? I'm operating at the level o=
f
installing apache and POSTing to it and CGI is the part that calls my c=
gi
script...

Thanks again
Chris


=

--0__=0ABBFCBDDFE3EDCD8f9e8a93df938690918c0ABBFCBDDFE3EDCD
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline
Content-transfer-encoding: quoted-printable

Nick Kew <nick@webthing.com> wrote on 09/11/2009 11:18:03 =
AM:

>

> Chris Cross wrote:

> > Nick Kew <nick@webthing.com> wrote on 09/11/2009 10:31:=
46 AM:

> >  > Hmmm.  That sounds buggish (dammit, it's a lo=
ng time since I did

> >  > anything nontrivial with CGI).

> >  >

> >  > What Apache version, and what CGI implementation?<=
br>
> >

> >  From my CGI env:

>

> That's not actually what I asked (many CGI implementations would r>
> give the same).  But it's probably enough to go on: a recent-=


> enough apache release, probably prefork, and probably mod_cgi

> (any Ubuntu folks here know whether it could be something

> different without the server admin having made a conscious choice)=
?

>

> Quick sanity checks:

>

> (1) if your test script just reads to EOF, does it get the right r>
>      (de-chunked) inputs?

> (2) does it fail with any reasonably-standard CGI library?

>

> --

> Nick Kew



Nick,



1) Reading till EOF works fine. I'd gotten in the habit of using Co=
ntent-Length in the non-chunked case so hadn't tried it until now. The =
"Transfer-Encoding: chunked" header is set. Is that proper if=
"de-chunking" has occurred?



2) I'm just doing C++ without any third party libraries.



So my test program is working now but I have another questions. Can=
you elaborate on what is a CGI implementation? I'm operating at the le=
vel of installing apache and POSTing to it and CGI is the part that cal=
ls my cgi script...



Thanks again

Chris







=

--0__=0ABBFCBDDFE3EDCD8f9e8a93df938690918c0ABBFCBDDFE3EDCD--

Re: How does Transfer-Encoding: chunked work withCGI?

Chris Cross wrote:

> Nick,
>
> 1) Reading till EOF works fine. I'd gotten in the habit of using
> Content-Length in the non-chunked case so hadn't tried it until now. The
> "Transfer-Encoding: chunked" header is set. Is that proper if
> "de-chunking" has occurred?

Well, the CGI spec pre-dates HTTP/1.1, and is silent on the subject
of chunked encoding. But now that you mention it, it seems to me
to make sense to remove the header when we de-chunk the input.

> So my test program is working now but I have another questions. Can you
> elaborate on what is a CGI implementation?

mod_cgi is one. mod_cgid is another. Then there are a couple
of fastcgi implementations including mod_fcgid. Then there are
language-specific implementations, such as mod_perl's, the first
to run in-process CGI.

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: How does Transfer-Encoding: chunked work withCGI?

Hi.
As regarding the overall content-length header, has anyone re-checked
the relevant part of the HTTP RFCs ? I haven't yet, but I seem to
remember that for chunked encoding, each chunk has an individual length,
but there is no overall content-length.
Which would meake sense, because at the time the initial headers are
sent out, the total length is not necessarily known.
In terms of CGI, that would mean that when Apache calls the CGI module,
it cannot set the CONTENT-LENGTH as an environment value, since it isn't
known yet.


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: How does Transfer-Encoding: chunked work withCGI?

André Warnier wrote:
> Hi.
> As regarding the overall content-length header, has anyone re-checked
> the relevant part of the HTTP RFCs ? I haven't yet, but I seem to
> remember that for chunked encoding, each chunk has an individual length,
> but there is no overall content-length.
> Which would meake sense, because at the time the initial headers are
> sent out, the total length is not necessarily known.
> In terms of CGI, that would mean that when Apache calls the CGI module,
> it cannot set the CONTENT-LENGTH as an environment value, since it isn't
> known yet.

That's what it does, yup. In fact the combination is prohibited, so the
content length header is unset in the presence of chunked encoding.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org