CPUs swamped by old httpd processes
am 15.11.2007 23:02:51 von Bennett HaseltonI have several dedicated servers running James Marshall's CGIProxy
script from www.jmarshall.com. My goal is to get them serving the
maximum number of hits per hour that the hardware is capable of, but
the sites are showing really high load averages, and high utilization
of the CPU by httpd processes that are really old (groups that are all
30 minutes old and 90 minutes old, in the examples of "top" output
from two servers, which I've pasted below).
Now, one possibility is that this is what I wanted anyway. If every
request takes a given amount of CPU, then if the CPU is pegged, and
it's not spending any time on busy-work like swapping (and I've ruled
that out -- "top" shows swap at 50K at most), it means the server is
handling the maximum amount of hits per hour that it can take. And
that despite the common sayings like "high load averages are bad",
"pegged CPU is bad", etc., nothing that I changed on the server would
enable it to handle more hits per hour anyway, short of replacing the
CPU.
But with load averages of 10, 60, or (on one server, really) 250, I
want to make sure my logic is sound before letting them chug along
like that! All those httpd processes using 10% of the CPU apiece that
have been running for 28 minutes, is it really just as important to
keep those running, as any other httpd processes? Or are they
probably useless at that point?
And here's the kicker, if those old processes really are useless, is
there some setting in httpd.conf to make old, CPU-expensive, probably
useless processes die out?
Here is the sample output of "top" from two of our servers.
[root@D3080 ~]# top
top - 10:03:38 up 51 days, 1:02, 1 user, load average: 10.11,
10.04, 9.94
Tasks: 62 total, 11 running, 51 sleeping, 0 stopped, 0 zombie
Cpu(s): 86.7% us, 13.3% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi,
0.0% si
Mem: 1019032k total, 312500k used, 706532k free, 18484k
buffers
Swap: 2031608k total, 15796k used, 2015812k free, 68712k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19006 apache 25 0 36828 24m 6472 R 10.3 2.5 28:41.62 httpd
19017 apache 25 0 40700 29m 6440 R 10.3 2.9 33:52.23 httpd
19008 apache 25 0 34108 23m 6060 R 10.0 2.3 34:31.75 httpd
19011 apache 25 0 36544 25m 6408 R 10.0 2.6 33:38.42 httpd
19014 apache 25 0 38600 27m 6484 R 10.0 2.8 27:09.78 httpd
19016 apache 25 0 36892 26m 6456 R 10.0 2.6 31:59.36 httpd
19018 apache 25 0 37000 26m 6464 R 10.0 2.6 32:10.96 httpd
19007 apache 25 0 40032 29m 6468 R 9.7 2.9 32:59.11 httpd
19013 apache 25 0 38188 27m 6560 R 9.7 2.8 32:20.71 httpd
19015 apache 25 0 39252 27m 5964 R 9.7 2.8 36:15.49 httpd
1 root 15 0 1740 284 256 S 0.0 0.0 0:23.07 init
2 root 34 19 0 0 0 S 0.0 0.0 0:04.08 ksoftirqd/
0
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/
0
4 root 10 -5 0 0 0 S 0.0 0.0 0:00.04 events/0
5 root 16 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 kthread
8 root 10 -5 0 0 0 S 0.0 0.0 1:14.44 kblockd/0
9 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
or this server where two instances of httpd were using 47% of the CPU
apiece:
[root@D3310 ~]# top
top - 10:07:02 up 6:57, 1 user, load average: 3.11, 3.03, 3.01
Tasks: 56 total, 4 running, 52 sleeping, 0 stopped, 0 zombie
Cpu(s): 85.0% us, 12.7% sy, 2.3% ni, 0.0% id, 0.0% wa, 0.0% hi,
0.0% si
Mem: 450916k total, 184088k used, 266828k free, 20080k
buffers
Swap: 917496k total, 9144k used, 908352k free, 79692k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7116 apache 25 0 40704 29m 5976 R 47.9 6.8 88:09.39 httpd
7122 apache 25 0 38724 29m 5944 R 46.9 6.6 96:29.47 httpd
4800 root 39 19 7244 1688 1260 R 2.3 0.4 10:44.03 http
10179 root 16 0 2428 932 760 R 0.3 0.2 0:00.01 top
1 root 16 0 2020 472 440 S 0.0 0.1 0:00.39 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/
0
3 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.01 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
18 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
39 root 14 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
19 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
38 root 15 0 0 0 0 S 0.0 0.0 0:03.13 kswapd0
185 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
296 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord
304 root 15 0 0 0 0 S 0.0 0.0 0:02.07 kjournald
1376 root 6 -10 3472 364 360 S 0.0 0.1 0:00.02 udevd
1570 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kauditd