strange random delays in a proxy configuration
am 23.11.2009 18:09:43 von awHi guys.
A customer of ours have an internal Apache 2.2 server under Linux
Debian. Static pages of their applications are located on their own server.
Some URLs are proxied by their server to our own Apache 2.2 server
(Linux Debian), through the Internet, by means of a line like
ProxyPass /getthumb http://our-server.company.com
and the URLs are like
/getthumb?id=(thumb-id)
These URLs obtain thumbnails from our server, corresponding to
documents. Each thumbnail is generated on the fly and is about 10 KB in
size. It takes our server on average 150 ms to process such a URL and
return the thumbnail to their server.
The number of thumbnails to serve for one customer application page is
variable, between just a few and sometimes several hundred, with an
average of 20-30 per page.
In the vast majority of cases, each thumbnail request is served in
around 150 ms, and we serve thousands such requests per day.
Yet sometimes, for no apparent reason that we can link to the process of
generating and returning the thumbnail, nor to any load of our own
Apache server or theirs, returning one arbitrary thumbnail in the series
takes about 30 seconds.
It is always either about 150 ms, or about 30000 ms. It is never
in-between, like 10 or 20 seconds or so.
The delay does not always affect the same thumbnails either. For the
same requested page with the same series of thumbnails, sometimes the 30
seconds delay will happen at the first and second thumbnail, sometimes 4
of them in the middle, and so on. Sometimes the whole page-worth of
thumbnails loads in exactly (number of thumbnails X 150 ms), which is
what it should always be.
If I watch our server's processes with "top", and simultaneously the end
of the access log or the screen of one of their workstations, I can see
nothing untoward when the browser suddenly "hangs" for 30 seconds
waiting fo the next thumbnail. The Apache processes on our side seem
mostly idle at that time, and there are plenty of Apache processes
available to serve any request.
It looks as if the browser on the customer side just hangs waiting for
the thumbnail, and our server is just idle.
These times are per our own server's access log, and per their Firefox
browser, to which we added the HttpFox add-on, to see what happens from
their side.
Their browsers do issue the requests (almost all at once), and the
symptoms are the same whether they use IE or Firefox.
My first question is : does there exist any tool in Apache itself, that
would show me what is happening during such a 30-second pause ?
The access log does not, as it just adds a line at the end of the 30
seconds, when the request is terminated, which is too late. There are
no errors in our error log.
mod_log_io and mod_forensic, from their documentation, do not seem to
provide much more relevant info.
For example, since their server proxies these calls to ours, and we see
nothing in our logs, is there a way whereby I can set the logging of the
proxy calls (and only those, preferably) to some "debug" setting ?
(I have full access to their and our Apache servers configurations)
My second question would be : does anyone have an idea of what could
explain sudden random 30 second pauses in such a schema ?
I am trying to avoid having to go through a TCP protocol analyser,
because I am unfamiliar with them, at least at the Linux level.
Thanks
------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org