Efficiently monitoring httpd children and signalling those using too much memory/cpu

--Apple-Mail-6-743555017
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii

We use Apache/mod_perl 2 and occasionally get a child httpd process that =
spins out of control, either consuming ever-increasing amounts of memory =
or max cpu. Usually due to an infinite loop or other bug in a specific =
part of the site - this sort of thing happens.

I would like to monitor for such httpd children every second or so, and =
when finding one, send it a USR2 signal so it can dump its current Perl =
stack to our error logs.

(Note that we are using rlimit/BSD::Resource, but when a process exceeds =
its memory rlimit, it dies immediately with no good way to log or take =
action (see Getting stack trace from Perl "Out of memory" error). So =
AFAICT rlimit will have to be complemented by an external monitoring =
program.)

I see that monit is highly regarded, but I can't figure out any way to =
monitor and send a signal to an individual httpd process. It seems =
geared towards killing or restarting services like apache as a whole. Am =
I missing something?

If I have to write a Perl script that does this, is there anything that =
will efficiently monitor the process table? Proc::ProcessTable takes .02 =
secs of cpu to generate a single table, so running that every second =
seems unnecessarily expensive. Opening up a "/usr/bin/top -b -d 1 |" and =
parsing the results is my current best idea.

Obviously would rather not badly reinvent a wheel if possible.

Thanks!
Jon


--Apple-Mail-6-743555017
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
charset=us-ascii

We use Apache/mod_perl 2 and occasionally get a child httpd process
that spins out of control, either consuming ever-increasing amounts of
memory or max cpu. Usually due to an infinite loop or other bug in a specific part of the site - this sort of thing happens.

I would like to monitor for such httpd children every second or so, and
when finding one, send it a USR2 signal so it can dump its current Perl
stack to our error logs.

(Note that we are using rlimit/BSD::Resource, but when a process
exceeds its memory rlimit, it dies immediately with no good way to log
or take action (see
is highly regarded, but I can't figure out any way to monitor and send a
signal to an individual httpd process. It seems geared towards killing
or restarting services like apache as a whole. Am I missing something?

If I have to write a Perl script that does this, is there anything that will efficiently monitor the process table?
takes .02 secs of cpu to generate a single table, so running that every
second seems unnecessarily expensive. Opening up a "/usr/bin/top -b -d 1
|" and parsing the results is my current best idea.

Obviously would rather not badly reinvent a wheel if possible.

Re: Efficiently monitoring httpd children and signalling those usingtoo much memory/cpu

--20cf307ca1a2f1647704ab23781e
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Aug 22, 2011 at 8:10 PM, Jonathan Swartz wrote:

> We use Apache/mod_perl 2 and occasionally get a child httpd process that
> spins out of control, either consuming ever-increasing amounts of memory or
> max cpu. Usually due to an infinite loop or other bug in a specific part of
> the site - this sort of thing happens.
>
> I would like to monitor for such httpd children every second or so, and
> when finding one, send it a USR2 signal so it can dump its current Perl
> stack to our error logs.
>
A few ideas:

- If your requests are typically short and the memory allocation uses
enough CPU time, you could set a soft limit for CPU time then catch
$SIG{XCPU} (you would also need to limit how many requests your child
processes handle). It worked for me in a quick test.
- If the memory usage is significant, as a quick check you could look at
the total free memory available on the system, and only if it falls below a
threshold do a more complex check with Proc::ProcessTable.
- If the runaway process causes the load average to go up, you could look
at the lod average, and only if it rises above a threshold do a more complex
check with Proc::ProcessTable.
- If your requests are typically short, you could create a small watchdog
server; a request would register its PID with the watchdog server, then
unregister when it finishes. If the watchdog sees a request register that
does not complete within some time limit, it could send SIGUSR2. I have
used a solution like this in the past, and it is effective, if a bit
cumbersome.
- Apache::Scoreboard
can
get you the PIDs of just the Apache processes, and some basic state
information. You might be able to use this to make your process table scan
more efficient. Maybe you could write a URL handler to do your checking
and signaling using the scoreboard from within Apache, then load the URL
periodically to trigger the test.

Hope this is helpful,

-----Scott.

--20cf307ca1a2f1647704ab23781e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Aug 22, 2011 at 8:10 PM, Jonathan Swartz < ref=3D"mailto:swartz@pobox.com">swartz@pobox.com> wrote:
<=
div class=3D"gmail_quote">

:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

We use Apache/mod_perl 2 and occasio=
nally get a child httpd process=20
that spins out of control, either consuming ever-increasing amounts of=20
memory or max cpu. Usually due to an infinite loop or other bug in a specif=
ic part of the site - this sort of thing happens.

I would like to mon=
itor for such httpd children every second or so, and=20
when finding one, send it a USR2 signal so it can dump its current Perl=20
stack to our error logs.

Segment Fault when creating a thread

This is a multi-part message in MIME format.

------=_NextPart_000_0028_01CC61B9.504ED750
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I must have a problem with my mod_perl set up. When I try to run this =
script, I get a "child pid 451 exit signal Segmentation fault (11)". It =
seems that creating the thread causes the error. This is the startup =
line from the log file with the apache, mod_perl, and perl versions.

Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming =
normal operations

The script that causes a failure is

#!/usr/bin/perl -wT

use strict;
use threads;
use threads::shared;

my $temp =3D threads->new(\&foo,"mike");
exit;

sub foo {
my $input =3D shift;
my $number =3D int(rand(10));
print "this went in the thread\n";
return($number);
}
------=_NextPart_000_0028_01CC61B9.504ED750
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



http-equiv=3DContent-Type>

Re: Segment Fault when creating a thread

Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK wrote:
> I must have a problem with my mod_perl set up.=A0 When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)".=A0 I=
t
> seems that creating the thread causes the error.=A0This is the startup li=
ne
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp =3D threads->new(\&foo,"mike");
> exit;
>
> sub foo {
>    my $input =3D shift;
>    my $number =3D int(rand(10));
>    print "this went in the thread\n";
>    return($number);
> }

Re: Segment Fault when creating a thread

Thanks for the tip Fred. I just now upgraded to 2.0.5 but still get the
same error.

Mike


----- Original Message -----
From: "Fred Moyer"
To: "Mike OK"
Cc: "mod_perl list"
Sent: Tuesday, August 23, 2011 5:26 PM
Subject: Re: Segment Fault when creating a thread


Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK wrote:
> I must have a problem with my mod_perl set up. When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
> seems that creating the thread causes the error. This is the startup line
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp = threads->new(\&foo,"mike");
> exit;
>
> sub foo {
> my $input = shift;
> my $number = int(rand(10));
> print "this went in the thread\n";
> return($number);
> }

Re: Segment Fault when creating a thread

I'm not that knowledgeable on threads, but this guy managed to get the
working with mpm_worker a month or so ago:

http://www.gossamer-threads.com/lists/modperl/modperl/103420

That is still a rather old version of Perl, so I would suggest trying
5.10 or something newer.

On Tue, Aug 23, 2011 at 2:58 PM, Mike OK wrote:
> Thanks for the tip Fred. =A0I just now upgraded to 2.0.5 but still get th=
e
> same error.
>
> Mike
>
>
> ----- Original Message ----- From: "Fred Moyer"
> To: "Mike OK"
> Cc: "mod_perl list"
> Sent: Tuesday, August 23, 2011 5:26 PM
> Subject: Re: Segment Fault when creating a thread
>
>
> Might want to try 2.0.5 which is about 6 months old.
>
> On Tue, Aug 23, 2011 at 2:23 PM, Mike OK wrote:
>>
>> I must have a problem with my mod_perl set up. When I try to run this
>> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
>> seems that creating the thread causes the error. This is the startup lin=
e
>> from the log file with the apache, mod_perl, and perl versions.
>>
>> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
>> normal operations
>>
>> The script that causes a failure is
>>
>> #!/usr/bin/perl -wT
>>
>> use strict;
>> use threads;
>> use threads::shared;
>>
>> my $temp =3D threads->new(\&foo,"mike");
>> exit;
>>
>> sub foo {
>> my $input =3D shift;
>> my $number =3D int(rand(10));
>> print "this went in the thread\n";
>> return($number);
>> }
>
>