Bookmarks

Yahoo Gmail Google Facebook Delicious Twitter Reddit Stumpleupon Myspace Digg

Search queries

wwwxxx,nvif, why atm producer might be held liable for economic injury, wwwxxxy=ServiceLogin, w2ksp4.exe, WwwxxXdbf, procmail "FROM_MAILER" patch, Use of assignment to $[ is deprecated at /usr/local/sbin/apxs line 86. , wwwxxx vim, mysql closing table and opening table, 800c5000

Links

XODOX
Impressum

#1: Efficiently monitoring httpd children and signalling those using too much memory/cpu

Posted on 2011-08-23 02:10:03 by Jonathan Swartz

--Apple-Mail-6-743555017
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii

We use Apache/mod_perl 2 and occasionally get a child httpd process that =
spins out of control, either consuming ever-increasing amounts of memory =
or max cpu. Usually due to an infinite loop or other bug in a specific =
part of the site - this sort of thing happens.

I would like to monitor for such httpd children every second or so, and =
when finding one, send it a USR2 signal so it can dump its current Perl =
stack to our error logs.

(Note that we are using rlimit/BSD::Resource, but when a process exceeds =
its memory rlimit, it dies immediately with no good way to log or take =
action (see Getting stack trace from Perl "Out of memory" error). So =
AFAICT rlimit will have to be complemented by an external monitoring =
program.)

I see that monit is highly regarded, but I can't figure out any way to =
monitor and send a signal to an individual httpd process. It seems =
geared towards killing or restarting services like apache as a whole. Am =
I missing something?

If I have to write a Perl script that does this, is there anything that =
will efficiently monitor the process table? Proc::ProcessTable takes .02 =
secs of cpu to generate a single table, so running that every second =
seems unnecessarily expensive. Opening up a "/usr/bin/top -b -d 1 |" and =
parsing the results is my current best idea.

Obviously would rather not badly reinvent a wheel if possible.

Thanks!
Jon


--Apple-Mail-6-743555017
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
charset=us-ascii

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><p>We use Apache/mod_perl 2 and occasionally get a child httpd process
that spins out of control, either consuming ever-increasing amounts of
memory or max cpu. Usually due to an infinite loop or other bug in a specific part of the site - this sort of thing happens.</p><p>I would like to monitor for such httpd children every second or so, and
when finding one, send it a USR2 signal so it can dump its current Perl
stack to our error logs.</p><p>(Note that we are using rlimit/BSD::Resource, but when a process
exceeds its memory rlimit, it dies immediately with no good way to log
or take action (see <a href="http://stackoverflow.com/questions/6348158">Getting stack trace from Perl "Out of memory" error</a>). So AFAICT rlimit will have to be complemented by an external monitoring program.)</p><p>I see that <a href="http://mmonit.com/monit/" rel="nofollow">monit</a>
is highly regarded, but I can't figure out any way to monitor and send a
signal to an individual httpd process. It seems geared towards killing
or restarting services like apache as a whole. Am I missing something?</p><p>If I have to write a Perl script that does this, is there anything that will efficiently monitor the process table? <a href="https://metacpan.org/release/Proc-ProcessTable" rel="nofollow">Proc::ProcessTable</a>
takes .02 secs of cpu to generate a single table, so running that every
second seems unnecessarily expensive. Opening up a "/usr/bin/top -b -d 1
|" and parsing the results is my current best idea.</p><p>Obviously would rather not badly reinvent a wheel if possible.</p><div>Thanks!</div><div>Jon</div><div><br></div></body></html>
--Apple-Mail-6-743555017--

Report this message

#2: Re: Efficiently monitoring httpd children and signalling those usingtoo much memory/cpu

Posted on 2011-08-23 05:05:08 by Scott Gifford

--20cf307ca1a2f1647704ab23781e
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Aug 22, 2011 at 8:10 PM, Jonathan Swartz <swartz@pobox.com> wrote:

> We use Apache/mod_perl 2 and occasionally get a child httpd process that
> spins out of control, either consuming ever-increasing amounts of memory or
> max cpu. Usually due to an infinite loop or other bug in a specific part of
> the site - this sort of thing happens.
>
> I would like to monitor for such httpd children every second or so, and
> when finding one, send it a USR2 signal so it can dump its current Perl
> stack to our error logs.
>
A few ideas:

- If your requests are typically short and the memory allocation uses
enough CPU time, you could set a soft limit for CPU time then catch
$SIG{XCPU} (you would also need to limit how many requests your child
processes handle). It worked for me in a quick test.
- If the memory usage is significant, as a quick check you could look at
the total free memory available on the system, and only if it falls below a
threshold do a more complex check with Proc::ProcessTable.
- If the runaway process causes the load average to go up, you could look
at the lod average, and only if it rises above a threshold do a more complex
check with Proc::ProcessTable.
- If your requests are typically short, you could create a small watchdog
server; a request would register its PID with the watchdog server, then
unregister when it finishes. If the watchdog sees a request register that
does not complete within some time limit, it could send SIGUSR2. I have
used a solution like this in the past, and it is effective, if a bit
cumbersome.
- Apache::Scoreboard<http://search.cpan.org/~mjh/Apache-Scoreboard-2.09.2/>
can
get you the PIDs of just the Apache processes, and some basic state
information. You might be able to use this to make your process table scan
more efficient. Maybe you could write a URL handler to do your checking
and signaling using the scoreboard from within Apache, then load the URL
periodically to trigger the test.

Hope this is helpful,

-----Scott.

--20cf307ca1a2f1647704ab23781e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Aug 22, 2011 at 8:10 PM, Jonathan Swartz <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:swartz@pobox.com">swartz@pobox.com</a>&gt;</span> wrote:<br><=
div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

<div style=3D"word-wrap:break-word"><p>We use Apache/mod_perl 2 and occasio=
nally get a child httpd process=20
that spins out of control, either consuming ever-increasing amounts of=20
memory or max cpu. Usually due to an infinite loop or other bug in a specif=
ic part of the site - this sort of thing happens.</p><p>I would like to mon=
itor for such httpd children every second or so, and=20
when finding one, send it a USR2 signal so it can dump its current Perl=20
stack to our error logs.</p></div></blockquote><div>A few ideas:</div><div>=
<ul><li>If your requests are typically short and the memory allocation uses=
enough CPU time, you could set a soft limit for CPU time then catch $SIG{X=
CPU} (you would also need to limit how many requests your child processes h=
andle). =A0It worked for me in a quick test.</li>

<li>If the memory usage is significant, as a quick check you could look at =
the total free memory available on the system, and only if it falls below a=
threshold do a more complex check with Proc::ProcessTable.</li><li>If the =
runaway process causes the load average to go up, you could look at the lod=
average, and only if it rises above a threshold do a more complex check wi=
th Proc::ProcessTable.</li>

<li>If your requests are typically short, you could create a small watchdog=
server; a request would register its PID with the watchdog server, then un=
register when it finishes. =A0If the watchdog sees a request register that =
does not complete within some time limit, it could send SIGUSR2. =A0I have =
used a solution like this in the past, and it is effective, if a bit cumber=
some.</li>

<li><a href=3D"http://search.cpan.org/~mjh/Apache-Scoreboard-2.09.2/">Apach=
e::Scoreboard</a>=A0can get you the PIDs of just the Apache processes, and =
some basic state information. =A0You might be able to use this to make your=
process table scan more efficient. =A0Maybe you could write a URL handler =
to do your checking and=A0signaling using the scoreboard from within Apache=
, then load the URL periodically to trigger the test.</li>

</ul><div>Hope this is helpful,</div><div><br></div><div>-----Scott.</div><=
div><br></div></div></div>

--20cf307ca1a2f1647704ab23781e--

Report this message

#3: Segment Fault when creating a thread

Posted on 2011-08-23 23:23:03 by Mike OK

This is a multi-part message in MIME format.

------=_NextPart_000_0028_01CC61B9.504ED750
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I must have a problem with my mod_perl set up. When I try to run this =
script, I get a "child pid 451 exit signal Segmentation fault (11)". It =
seems that creating the thread causes the error. This is the startup =
line from the log file with the apache, mod_perl, and perl versions.

Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming =
normal operations

The script that causes a failure is

#!/usr/bin/perl -wT

use strict;
use threads;
use threads::shared;

my $temp =3D threads->new(\&foo,"mike");
exit;

sub foo {
my $input =3D shift;
my $number =3D int(rand(10));
print "this went in the thread\n";
return($number);
}
------=_NextPart_000_0028_01CC61B9.504ED750
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.19120">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2 face=3DArial>I must have a problem with my mod_perl =
set=20
up.&nbsp; When I try to run this script, I get a "child pid 451 exit =
signal=20
Segmentation fault (11)".&nbsp; It seems that creating the thread causes =
the=20
error.&nbsp;This is the startup line from the log file with the apache,=20
mod_perl, and perl versions.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>Apache/2.2.11 (Unix) mod_perl/2.0.4 =
Perl/v5.8.5=20
configured -- resuming normal operations</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>The script that causes a failure =
is</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>#!/usr/bin/perl -wT</FONT></DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>use strict;<BR>use threads;<BR>use=20
threads::shared;</FONT></DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>my $temp =
threads-&gt;new(\&amp;foo,"mike");<BR></FONT><FONT size=3D2=20
face=3DArial>exit;</FONT></DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>sub foo {<BR>&nbsp;&nbsp; my $input =3D =

shift;<BR>&nbsp;&nbsp; my $number =3D int(rand(10));<BR>&nbsp;&nbsp; =
print "this=20
went in the thread\n";<BR>&nbsp;&nbsp;=20
return($number);<BR>}</FONT></DIV></BODY></HTML>

------=_NextPart_000_0028_01CC61B9.504ED750--

Report this message

#4: Re: Segment Fault when creating a thread

Posted on 2011-08-23 23:26:25 by Fred Moyer

Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mike@acorg.com> wrote:
> I must have a problem with my mod_perl set up.=A0 When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)".=A0 I=
t
> seems that creating the thread causes the error.=A0This is the startup li=
ne
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp =3D threads->new(\&foo,"mike");
> exit;
>
> sub foo {
>    my $input =3D shift;
>    my $number =3D int(rand(10));
>    print "this went in the thread\n";
>    return($number);
> }

Report this message

#5: Re: Segment Fault when creating a thread

Posted on 2011-08-23 23:58:22 by Mike OK

Thanks for the tip Fred. I just now upgraded to 2.0.5 but still get the
same error.

Mike


----- Original Message -----
From: "Fred Moyer" <fred@redhotpenguin.com>
To: "Mike OK" <mike@acorg.com>
Cc: "mod_perl list" <modperl@perl.apache.org>
Sent: Tuesday, August 23, 2011 5:26 PM
Subject: Re: Segment Fault when creating a thread


Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mike@acorg.com> wrote:
> I must have a problem with my mod_perl set up. When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
> seems that creating the thread causes the error. This is the startup line
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp = threads->new(\&foo,"mike");
> exit;
>
> sub foo {
> my $input = shift;
> my $number = int(rand(10));
> print "this went in the thread\n";
> return($number);
> }

Report this message

#6: Re: Segment Fault when creating a thread

Posted on 2011-08-24 00:15:19 by Fred Moyer

I'm not that knowledgeable on threads, but this guy managed to get the
working with mpm_worker a month or so ago:

http://www.gossamer-threads.com/lists/modperl/modperl/103420

That is still a rather old version of Perl, so I would suggest trying
5.10 or something newer.

On Tue, Aug 23, 2011 at 2:58 PM, Mike OK <mike@acorg.com> wrote:
> Thanks for the tip Fred. =A0I just now upgraded to 2.0.5 but still get th=
e
> same error.
>
> Mike
>
>
> ----- Original Message ----- From: "Fred Moyer" <fred@redhotpenguin.com>
> To: "Mike OK" <mike@acorg.com>
> Cc: "mod_perl list" <modperl@perl.apache.org>
> Sent: Tuesday, August 23, 2011 5:26 PM
> Subject: Re: Segment Fault when creating a thread
>
>
> Might want to try 2.0.5 which is about 6 months old.
>
> On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mike@acorg.com> wrote:
>>
>> I must have a problem with my mod_perl set up. When I try to run this
>> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
>> seems that creating the thread causes the error. This is the startup lin=
e
>> from the log file with the apache, mod_perl, and perl versions.
>>
>> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
>> normal operations
>>
>> The script that causes a failure is
>>
>> #!/usr/bin/perl -wT
>>
>> use strict;
>> use threads;
>> use threads::shared;
>>
>> my $temp =3D threads->new(\&foo,"mike");
>> exit;
>>
>> sub foo {
>> my $input =3D shift;
>> my $number =3D int(rand(10));
>> print "this went in the thread\n";
>> return($number);
>> }
>
>

Report this message