Possible infinite loop in mod_proxy

I apologize if this is not the correct place to ask questions about
potential mod_proxy bugs. It seemed the best choice, so please correct
me (gently) if I am wrong.

I have been working with mod_proxy under Apache 2.0.44 and have been
having some problems.

The basic setup is as follows:

Listen 8008

ServerName myproxy.mydomain.mytld
DocumentRoot "/unnecessary/path/to/files"

ProxyRequests On
ProxyPreserveHost on
ProxyReceiveBufferSize 4096
ProxyTimeout 20
ProxyVia off


In my browser, I set the proxy to be myproxy.mydomain.mytld at port
8008. Things seem to work all right for simple requests. However,
requesting a really messy page like "www.cnn.com" causes some
interesting problems. (For bug duplication purposes, I am currently
using Mozilla 1.3a, but this has been duplicated on various versions of
IE, as well).

After making the request to www.cnn.com, Apache must be forcefully
restarted in order serve any other requests reliably. Request threads
do not appear to want to give up their connection to the internet once
the request is completed. An ethereal dump indicates that Apache is
still sending TCP traffic to the destination server. An endless stream
of ACK packets is sent to the destination server until Apache is
forcefully restarted. I have verified that it is not the browser that
is doing this.

Of course, since the backend continues to think it is speaking with a
server, the frontend socket gets tied up and requests to other pages
block until the server is restarted, causing a host of other symptoms.
I believe that the rest of the problems I am experiencing are entirely
due to this seeming infinite loop.

Apache 2.0 is a complex piece of software, so I apologize if this is in
fact a bug in the server core rather than in mod_proxy. The server
seems to work all right without serving as a proxy, so it seemed
reasonable to hit this list.

C

Re: Possible infinite loop in mod_proxy

I have more information on this apparent bug.

As I watch the requests move around, I noticed that when making the
request through Squid, the connection: keep-alive and other relevant
headers are set correctly. When making the request through mod_proxy,
those headers disappear.

So, here is my take on this. I got mod_proxy to at least stop hanging
and constantly hitting the origin server by adding the following two
lines of code to mod_proxy.c:

origin->keep_alive = 0;
apr_table_set( r->headers_in, "Connection", "Close );

This was done in the middle of the ap_proxy_http_request function in
proxy_http.c.

This is, of course, not the right thing to do, but I wanted to see if it
would fix the problem, and it did. I can now correctly proxy, even if
it is slow because of the lack of keep-alive. I believe what is
happening is mod_proxy_http is deciding to keep the connection alive on
the backend, which is correct, but it is failing to inform the browser
that it has done so. In the absence of information about keep-alive,
the browser does not make any more requests on that socket, but also
fails to open more of them, since it has a limit as to how many it can
open, and the socket is not yet closed.

I am no expert on the nuances of when to send keep-alive headers and
when not to, but it is clear that something is missing here.

I hope this helps. Please let me know if more details are needed.

C

> In my browser, I set the proxy to be myproxy.mydomain.mytld at port
> 8008. Things seem to work all right for simple requests. However,
> requesting a really messy page like "www.cnn.com" causes some
> interesting problems. (For bug duplication purposes, I am currently
> using Mozilla 1.3a, but this has been duplicated on various versions
> of IE, as well).
>
> After making the request to www.cnn.com, Apache must be forcefully
> restarted in order serve any other requests reliably. Request threads
> do not appear to want to give up their connection to the internet once
> the request is completed. An ethereal dump indicates that Apache is
> still sending TCP traffic to the destination server. An endless
> stream of ACK packets is sent to the destination server until Apache
> is forcefully restarted. I have verified that it is not the browser
> that is doing this.

Re: Possible infinite loop in mod_proxy

Chris Monson wrote:

> I apologize if this is not the correct place to ask questions about
> potential mod_proxy bugs. It seemed the best choice, so please correct
> me (gently) if I am wrong.

Please repost this to the main dev list - this looks like it could be a
problem within the filters code, which the proxy makes extensive use of.

Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight..."

Re: Possible infinite loop in mod_proxy

Chris Monson wrote:

> So, here is my take on this. I got mod_proxy to at least stop hanging
> and constantly hitting the origin server by adding the following two
> lines of code to mod_proxy.c:
>
> origin->keep_alive = 0;
> apr_table_set( r->headers_in, "Connection", "Close );

What you have effectively done is turn keepalives off.

> This is, of course, not the right thing to do, but I wanted to see if it
> would fix the problem, and it did. I can now correctly proxy, even if
> it is slow because of the lack of keep-alive. I believe what is
> happening is mod_proxy_http is deciding to keep the connection alive on
> the backend, which is correct, but it is failing to inform the browser
> that it has done so.

Keepalive is implied by the HTTP/1.1 protocol, there is no need for
proxy to inform the browser.

It would help if you attach a debugger to the stuck process and see
where it is getting stuck in the infinite loop.

Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight..."

Re: Possible infinite loop in mod_proxy

>
> Please repost this to the main dev list - this looks like it could be
> a problem within the filters code, which the proxy makes extensive use
> of.


Thanks, I will do that, along with attaching a debugger as you suggested.

C