mod_proxy intermittent "(70007)The timeout specified has expired"

Hi,

I have two web servers that proxy around 150 backend sites, and are
experiencing a lot of mod_proxy timeout errors. The two servers are Xen
guests running CentOS 5.3, and sit behind an LVS-NAT load balancer which
is also a Xen domU, also running CentOS 5.3. The errors in the error log
are:

[error] (70007)The timeout specified has expired: proxy: HTTP: attempt
to connect to x.x.x.x:80 (www.somesite.com) failed
[error] ap_proxy_connect_backend disabling worker for (www.somesite.com)
[error] proxy: HTTP: disabled connection for (www.somesite.com)

The last error above appears every time a request is made for
www.somesite.com until that backend site is re-enabled by mod_proxy. The
errors affect a number of different backend sites with the only common
factor appearing to be that these are among the most popular
reverse-proxied sites. The errors don't seem to appear for the less
popular backend sites.

Something I've observed a number of times is that a TCP connection to a
particular backend site will be stuck in the SYN_SENT state for a long
time (approx. 2 minutes), and as soon as that connection disappears the
timeout error occurs and the backend site is disabled.

Until last week this service was running on a single domU with no load
balancer, on completely different hardware. In this configuration the
timeout errors had occurred however they were nowhere near as consistent
- about 2 or 3 times a month compared to at least once per hour that
we're experiencing now.

We receive an average of around 300,000 hits per hour, up to about a
million during busy parts of the day (as reported by AWStats). We have a
10Mbps internet link however our cacti graphs show the average usage as
being around 2Mbps, peaking to 3Mbps, so it's unlikely that we're
saturating the network link.

Here is an example virtual host directive from our configuration files:

--------

ServerAdmin admin@ourcompany.com
DocumentRoot /var/www/html
ServerName "www-somesite-com.ourcompany.com"
LogLevel warn
ServerSignature Off

SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:+MEDIUM:!EXP:!SSLv2:!LOW
SSLCertificateKeyFile /etc/httpd/certs/sp.key
SSLCertificateFile /etc/httpd/certs/sp.cer
SSLCertificateChainFile /etc/httpd/certs/server-chain.crt

ProxyPass / http://www.somesite.com/
ProxyPassReverse / http://www.somesite.com/
ProxyHTMLURLMap http://www.somesite.com /

SetOutputFilter line-editor
RequestHeader unset Accept-Encoding
ProxyPassReverseCookieDomain .somesite.com .ourcompany.com

SetEnv LineEdit
"application/x-javascript;text/plain;text/css;text/javascrip t;text/html"
LERewriteRule "\"http\://www\.somesite\.com/" "\"/" R
LERewriteRule "\'http\://www\.somesite\.com/" "'/" R

Include conf.d/shib.conf.include

ErrorDocument 502
https://sp.ourcompany.com/errors/providerSiteError_502.html

SetEnv proxy-nokeepalive 1


--------

Here are some more details of the web server:

# uptime
14:04:05 up 5:28, 1 user, load average: 0.00, 0.01, 0.02

# uname -a
Linux ckn-sp-2 2.6.18-128.4.1.el5xen #1 SMP Tue Aug 4 20:51:12 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux

# free
total used free shared buffers cached
Mem: 4194304 946272 3248032 0 32312 127636
-/+ buffers/cache: 786324 3407980
Swap: 2031608 0 2031608

# /usr/sbin/httpd -v
Server version: Apache/2.2.3
Server built: Jul 14 2009 06:02:39

# netstat -s
Ip:
2934756 total packets received
0 forwarded
0 incoming packets discarded
2934742 incoming packets delivered
3207218 requests sent out
Icmp:
2267 ICMP messages received
10 input ICMP message failed.
ICMP input histogram:
destination unreachable: 286
redirects: 1516
echo requests: 465
773 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 308
echo replies: 465
IcmpMsg:
InType3: 286
InType5: 1516
InType8: 465
OutType0: 465
OutType3: 308
Tcp:
61435 active connections openings
110304 passive connection openings
36 failed connection attempts
2578 connection resets received
86 connections established
2777064 segments received
3011011 segments send out
40041 segments retransmited
0 bad segments received.
318 resets sent
Udp:
157061 packets received
264 packets to unknown port received.
0 packet receive errors
157337 packets sent
TcpExt:
3 invalid SYN cookies received
36 resets received for embryonic SYN_RECV sockets
39 packets pruned from receive queue because of socket buffer overrun
2 ICMP packets dropped because socket was locked
95627 TCP sockets finished time wait in fast timer
735 time wait sockets recycled by time stamp
53736 delayed acks sent
28 delayed acks further delayed because of locked socket
Quick ack mode was activated 1415 times
37023 packets directly queued to recvmsg prequeue.
1568 packets directly received from backlog
414402 packets directly received from prequeue
835894 packets header predicted
25303 packets header predicted and directly queued to user
641977 acknowledgments not containing data received
491841 predicted acknowledgments
653 times recovered from packet loss due to fast retransmit
129 times recovered from packet loss due to SACK data
11 congestion windows fully recovered
TCPDSACKUndo: 39
35 congestion windows recovered after partial ack
44 TCP data loss events
104 timeouts after reno fast retransmit
29 timeouts after SACK recovery
8 timeouts in loss state
1044 fast retransmits
22 forward retransmits
779 retransmits in slow start
5787 other TCP timeouts
TCPRenoRecoveryFail: 19
3 sack retransmits failed
1 times receiver scheduled too late for direct processing
1744 packets collapsed in receive queue due to low socket buffer
1227 DSACKs sent for old packets
144 DSACKs received
124 connections reset due to unexpected data
689 connections reset due to early user close
3991 connections aborted due to timeout
10 times unabled to send RST due to no memory
IpExt:


----

All ideas are welcome.

Thanks,
Nick

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org