Re: Runaway Apache Process

On Wed, Jan 27, 2010 at 8:28 AM, Dan Bunyard wrote:
> I have never done a backtrace, can you please point me in the right
> direction for that?

gdb /path/to/httpd PID
....
(gdb) where
(backtrace displayed here)
(gdb) quit

run gdb as root if you start httpd as root

> I didn't check CPU usage at the time, only load average which was around =
100
> (normally it's between 0.02 and 0.5 over 1 minute).

top might give a good clue

> I was able to log in but it was VERY slow.=A0 As I watched the load avera=
ge it
> was continuing to climb just before I killed Apache.=A0 It did not termin=
ate
> gracefully either, the error_log showed this:
....

> [Mon Jan 25 12:50:49 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:55 2010] [error] child process 23437 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23441 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23445 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23451 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23453 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 28350 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 28355 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 26939 still did not exit=
,
> sending a SIGKILL
> [Mon Jan 25 12:50:56 2010] [notice] caught SIGTERM, shutting down

hmmm.... far more "[warn] ... sending a SIGTERM" than "[error] ...
sending a SIGKILL"... I guess that means the processes in
interruptible code and system performance is degraded enough that it
takes a long time for the processes to get cleaned up. (I doubt
there's anything to learn there.)

> Is there a way to turn on more logging (debug logs) or a better way to tr=
ace
> what it doing at that time?

no httpd debug logs would help AFAIK; no clue here what PHP offers

you could pick one of the high CPU processes and strace it to see
which, if any, system calls are being issued during that time

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org