POST request with chunked encoding

am 19.01.2005 12:59:44 von Erik Knudsen

Hi!

I'm experiencing problems with creating a chunked post request to IIS.

Scenario:

Client side:
---------------

Server side:
----------------

(The content-length is max int, and totally wrong...)

How can the CGI program know when the stream changes from "not chunked"
to "chunked"?

If there is a more appropriate newsgroup for this question, feel free to
tell me:-)

Regards,
Erik Knudsen

Re: POST request with chunked encoding

am 21.01.2005 04:47:33 von wadeh

Hi Erik,

What version (and service pack) of IIS are you using?

The 0xffffffff value in the content-length header is by design (although
probably not documented well.) IIS sets this when the request is chunked
transfer encoded.

The appearance of chunk encoding in the entity stream sounds like something
that we saw in ISAPI at some point in the past. When a request arrives with
an entity body, IIS core will read in some of the data before calling into
user code - by default, this will be 48k of data. At some point (I don't
remember it is was IIS 4 or something in IIS 5 that got fixed after it
released), IIS was dechunking the pre-read entity, but not removing the
encoding from the stream after that. This is probably why you don't see it
for a while and then, once you get past the pre-read buffer, you see the raw
stream.

If my hypothesis is correct, you can work around the problem in two ways:
First, you could increase the UploadReadAheadSize setting to something
bigger than the data that you expect to receive. Then, you will not see any
chunk encoding in the request. Of course, the downside is that if you
receive a bigger request than you expect, then you'll have problems.
Alternately, you could set UploadReadAheadSize to zero. In this case, IIS
will not buffer any of the request entity, and you will see - and be able to
reliably parse out - the chunk encoding.

Here is the MSDN topic on UploadReadAheadSize:

http://msdn.microsoft.com/library/default.asp?url=/library/e n-us/iissdk/html/8b7978ef-2034-4301-a432-29bc90508e8c.asp

I hope that this helps,
-Wade A. Hilmo,
-Microsoft

PS: If you have any questions on CGI issues, you could try posting them to
the microsoft.public.platformsdk.internet.server.isapi-dev newsgroup. It's
a bit off-topic, but the Microsoft IIS folks that hang out there are
generally familiar with CGI, too.

"Erik Knudsen" wrote in message
news:OI4Md5h$EHA.4004@tk2msftngp13.phx.gbl...
> Hi!
>
> I'm experiencing problems with creating a chunked post request to IIS.
>
> Scenario:
>
> Client side:
> ---------------
>
>
>
>
> Server side:
> ----------------
>
>
> (The content-length is max int, and totally wrong...)
>
>
>
>
> How can the CGI program know when the stream changes from "not chunked"
> to "chunked"?
>
> If there is a more appropriate newsgroup for this question, feel free to
> tell me:-)
>
>
> Regards,
> Erik Knudsen

Re: POST request with chunked encoding

am 21.01.2005 11:12:32 von Erik Knudsen

Wade A. Hilmo [MS] wrote:
> Hi Erik,
>
> What version (and service pack) of IIS are you using?

I'm using IIS 5.1 on Windows XP SP2 with all service packs and patches
from windowsupdate.

> The 0xffffffff value in the content-length header is by design (although
> probably not documented well.) IIS sets this when the request is chunked
> transfer encoded.

We suspected that to be the case, but the stream was de-chunked. Kind of
misleading:-)

> The appearance of chunk encoding in the entity stream sounds like something
> that we saw in ISAPI at some point in the past. When a request arrives with
> an entity body, IIS core will read in some of the data before calling into
> user code - by default, this will be 48k of data. At some point (I don't
> remember it is was IIS 4 or something in IIS 5 that got fixed after it
> released), IIS was dechunking the pre-read entity, but not removing the
> encoding from the stream after that. This is probably why you don't see it
> for a while and then, once you get past the pre-read buffer, you see the raw
> stream.

We suspected something along these lines - thanks for the conformation:-)

It doesn't seem to be fixed in 5.1 - but maybe I'm missing some patch or
update that I don't get from regular windowsupdate.

> If my hypothesis is correct, you can work around the problem in two ways:
> First, you could increase the UploadReadAheadSize setting to something
> bigger than the data that you expect to receive. Then, you will not see any
> chunk encoding in the request. Of course, the downside is that if you
> receive a bigger request than you expect, then you'll have problems.
> Alternately, you could set UploadReadAheadSize to zero. In this case, IIS
> will not buffer any of the request entity, and you will see - and be able to
> reliably parse out - the chunk encoding.

I guess setting the UploadReadAheadSize to 0 would be the safest choice,
especially since IIS still sets the content-length to 0xffffffff. But I
don't like depending on our clients to be aware of this setting and this
problem, or downloading obscure patches other than the windowsupdate
ones, so we might end up with streming the data using several smaller
http posts, hoping that the UploadReadAheadSize setting will always be
either above our "packet" size of a few K, or 0.

> PS: If you have any questions on CGI issues, you could try posting them to
> the microsoft.public.platformsdk.internet.server.isapi-dev newsgroup. It's
> a bit off-topic, but the Microsoft IIS folks that hang out there are
> generally familiar with CGI, too.

Ok, thanks for all the info and help! I was kind of confused about this
issue:-)

Regards,
Erik Knudsen

Re: POST request with chunked encoding

am 21.01.2005 11:12:48 von Erik Knudsen