apache2 processes hanging over a few hours until MaxClients isreached

apache2 processes hanging over a few hours until MaxClients isreached

am 12.11.2009 08:54:37 von Steve Cox

Hi,

I have an issue that overnight, my server's apache processes seem to get lo=
cked and stop responding to users. They have to close browsers sessions and=
start again. Meanwhile, the server is creating more apache processes - whi=
ch in turn are eventually locked out too. In the end, MaxClients is reached=
..

The server used is public but is used by company staff for a CRM database -=
connections at the time of failure are from company office IPs so it doesn=
't look like intentional DoS.

The server is predominantly running PHP but I can't see why the situation o=
ccurs at roughly the same time each night so I am looking at apache first.

To see an example of how the process usage escalates, see:

http://i823.photobucket.com/albums/zz153/g00seUK/trowselocal domain-apache_p=
rocesses-.png

When it occurred yesterday, MaxClients had been reached. Today, apache was =
reset before that number had been reached. But you can still see the same p=
ath.


Apache 2.0.55
Linux kernel 2.6.15
PHP 5.1.2

When maxed out, all apache processes are in CLOSE_WAIT state. Prior to this=
, the spare servers are in LISTEN but a number of the apache processes are =
CLOSE_WAIT.

eg: netstat:

tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:16795 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:8346 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:27024 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:31721 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:31476 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:22722 TIME_WA=
IT -
tcp6 0 0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:12247 TIME_WA=
IT -
tcp6 38 0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2791 CLOSE_W=
AIT 13243/apache2
tcp6 38 0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2050 CLOSE_W=
AIT 4788/apache2
tcp6 38 0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:1282 CLOSE_W=
AIT 4789/apache2
tcp6 38 0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2064 CLOSE_W=
AIT 4791/apache2

At the time this is occurring, I used gdb against some of the CLOSE_WAIT pr=
ocesses and got:


(gdb) thread apply all bt full

Thread 1 (Thread 46912528496000 (LWP 12316)):
#0 0x00002aaaac1009a9 in flock () from /lib/libc.so.6
No symbol table info available.
#1 0x00002aaaacf248e0 in zif_session_is_registered () from /usr/lib/apache=
2/modules/libphp5.so
No symbol table info available.
#2 0x00002aaaacf24a57 in ps_read_files () from /usr/lib/apache2/modules/li=
bphp5.so
No symbol table info available.
#3 0x00002aaaacf2220f in php_session_start () from /usr/lib/apache2/module=
s/libphp5.so
No symbol table info available.
#4 0x00002aaaacf229c9 in zif_session_start () from /usr/lib/apache2/module=
s/libphp5.so
No symbol table info available.
#5 0x00002aaaad04e125 in execute () from /usr/lib/apache2/modules/libphp5.=
so
No symbol table info available.
#6 0x00002aaaad04d813 in execute () from /usr/lib/apache2/modules/libphp5.=
so
No symbol table info available.
#7 0x00002aaaad032629 in zend_execute_scripts () from /usr/lib/apache2/mod=
ules/libphp5.so
No symbol table info available.
#8 0x00002aaaacff4dda in php_execute_script () from /usr/lib/apache2/modul=
es/libphp5.so
No symbol table info available.
#9 0x00002aaaad0c188f in zend_get_zval_ptr () from /usr/lib/apache2/module=
s/libphp5.so
No symbol table info available.
#10 0x00000000004364f5 in ap_run_handler ()
No symbol table info available.
#11 0x0000000000436952 in ap_invoke_handler ()
No symbol table info available.
#12 0x00000000004288a0 in ap_process_request ()
No symbol table info available.
#13 0x00000000004238a8 in _start ()
No symbol table info available.
#0 0x00002aaaac1009a9 in flock () from /lib/libc.so.6

I did not run it against all the processes - but the ones I did were identi=
cal.

More info:

Compiled in modules:
core.c
mod_access.c
mod_auth.c
mod_log_config.c
mod_logio.c
mod_env.c
mod_setenvif.c
prefork.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_negotiation.c
mod_dir.c
mod_alias.c
mod_so.c

By config:

cgi, deflate, expires, mod-security, php5, rewrite, ssl, userdir.

Many thanks,
Steve

This email was sent by a company owned by Pearson plc, registered office at=
80 Strand, London WC2R 0RL.
Registered in England and Wales with company number 53723


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org