Killing oversized Perl processes
Killing oversized Perl processes
am 11.03.2010 22:41:38 von ArthurG
--Apple-Mail-1-126206485
Content-Type: text/plain;
charset=US-ASCII;
format=flowed;
delsp=yes
Content-Transfer-Encoding: 7bit
Running Perl programs in mod_perl in Apache (2.2) on RHEL:
> [10 artg@virtualplant:/etc]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
> [11 artg@virtualplant:/etc]$ uname -r
> 2.6.18-164.11.1.el5
Occasionally a process grows so large that it freezes the system:
> several of them will use so much memory that kswapd takes all the CPU:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
and
> from /etc/httpd/logs/error_log
> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing
> the http daemon from connecting to network port 3306 For complete
> SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.
> Feb 24 14:55:06 virtualplant last message repeated 6 times
> Feb 24 15:00:44 virtualplant last message repeated 3 times
> Feb 24 15:00:55 virtualplant last message repeated 5 times
> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to
> 128.122.128.24 port 67
> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked oom-
> killer: gfp_mask=0xd0, order=0, oomkilladj=0
> Feb 24 15:09:51 virtualplant kernel:
> Feb 24 15:09:51 virtualplant kernel: Call Trace:
> Feb 24 15:09:51 virtualplant kernel: []
> out_of_memory+0x8e/0x2f3
> Feb 24 15:09:51 virtualplant kernel: []
> __alloc_pages+0x245/0x2ce
> Feb 24 15:09:51 virtualplant kernel: [] cache_grow
> +0x133/0x3c1
> Feb 24 15:09:51 virtualplant kernel: []
> cache_alloc_refill+0x136/0x186
> Feb 24 15:09:51 virtualplant kernel: []
> kmem_cache_alloc+0x6c/0x76
> Feb 24 15:09:51 virtualplant kernel: [] getname
> +0x25/0x1c2
> Feb 24 15:09:51 virtualplant kernel: []
> do_sys_open+0x17/0xbe
> Feb 24 15:09:51 virtualplant kernel: [] tracesys
> +0xd5/0xe0
Then I need to cycle the box's power.
I'm implementing a multi-layer defense against this.
1) Try to prevent input that might blow up a process. However, this
will be imperfect.
2) Kill apache httpd processes occasionally, to control the effect of
slow perl memory leaks. I'll do this by setting MPM Worker
MaxRequestsPerChild to some modest value. (I'll try 100.)
3) Kill processes that grow too big, which concerns this message.
In bash, ulimit sets user resource limits. With mod_perl on Apache
Apache2::Resource controls the size of httpd processes. Both
eventually call setrlimit(int resource, const struct rlimit *rlim).
With Apache2::Resource one can put this in the httpd.conf:
> PerlModule Apache2::Resource
> # set child memory limit in megabytes
> # RLIMIT_AS (address space) will work to limit the size of a process
> on Linux
> PerlSetEnv PERL_RLIMIT_AS 1000:1100
> # this loads Apache2::Resource for each new httpd; that will set the
> ulimits from the Perl environment variables
> PerlChildInitHandler Apache2::Resource
OK, that kills big processes. What happens next is that Perl runs out
of memory (outputs "Out of Memory!") and calls the __DIE__ signal
handler. So, my plan is to catch the signal, redirect the browser to
an error page, and finally kill the process. Before the http Request
handler is called I say:
> $SIG{__DIE__} = \&oversizedHandler;
>
Then when __DIE__ fires the code below runs.
> use CGI;
> use English;
> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>
> # SIG handler called when __DIE__ fires
> sub oversizedHandler{
> my $a = shift;
> chomp $a;
> print STDERR "handler in process $PID called with '$a'\n";
>
> # up the soft AS limit to the hard limit, so that we've some RAM
> to use; in this example we free up 100 MB, much more than needed
> my $success = setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100
> * 1024 * 1024);
> if( $success ) {
> print STDERR "set limits to 512*1024 * 1024\n";
> }
>
> $cgi = CGI->new;
> print $cgi->redirect( -location => 'http://website.com/program.cgi¶m1=value1¶m2=value2)
> ;
>
> CORE::exit();
> }
>
Here's the problem. Nothing goes to STDOUT, so I cannot write to the
browser.
Thus, my question is: how can one kill an oversized process and
provide feedback to the user at the browser?
One alternative seems to be to use the special variable $^M (see the
perlvar manpage for more details) as recommended by the modperlbook.
How does one determine whether -DPERL_EMERGENCY_SBRK is defined, and
if one does that does STDOUT still work?
BR
A
Arthur P. Goldberg
Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org
Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences
www.cs.nyu.edu/artg
artg@cs.nyu.edu
New York University
212 995-4918
Coruzzi Lab
8th Floor Silver Building
1009 Silver Center
100 Washington Sq East
New York NY 10003-6688
--Apple-Mail-1-126206485
Content-Type: text/html;
charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
-webkit-line-break: after-white-space; ">Running Perl programs =
in mod_perl in Apache (2.2) on =
RHEL:
class=3D"Apple-style-span" color=3D"#000000">[10 =
artg@virtualplant:/etc]$ cat /etc/redhat-release
Red Hat Enterprise =
Linux Server release 5.4 (Tikanga)
[11 artg@virtualplant:/etc]$ uname =
-r
2.6.18-164.11.1.el5
v>
Occasionally a process grows so large that it freezes =
the system:
class=3D"Apple-style-span" color=3D"#000000">several of them will use so =
much memory that kswapd takes all the CPU:
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: 12px;"> PID =
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
349 =
root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
348 root 20 =
-5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: =
12px;">
and
align=3D"">
color=3D"#000000">from /etc/httpd/logs/error_log
Feb
24 14:35:32 virtualplant setroubleshoot: SELinux is preventing the http
daemon from connecting to network port 3306 For complete SELinux
messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
Feb 24 =
14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.
Feb =
24 14:55:06 virtualplant last message repeated 6 times
Feb 24 =
15:00:44 virtualplant last message repeated 3 times
Feb 24 15:00:55 =
virtualplant last message repeated 5 times
Feb 24 15:01:21 =
virtualplant dhclient: DHCPREQUEST on eth0 to 128.122.128.24 port =
67
Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0
Feb 24 =
15:09:51 virtualplant kernel:
Feb 24 15:09:51 virtualplant kernel: =
Call Trace:
Feb 24 15:09:51 virtualplant kernel: =
[<ffffffff800c6076>] out_of_memory+0x8e/0x2f3
Feb 24 15:09:51 =
virtualplant kernel: [<ffffffff8000f487>] =
__alloc_pages+0x245/0x2ce
Feb 24 15:09:51 virtualplant kernel: =
[<ffffffff80017812>] cache_grow+0x133/0x3c1
Feb 24 15:09:51 =
virtualplant kernel: [<ffffffff8005c2e5>] =
cache_alloc_refill+0x136/0x186
Feb 24 15:09:51 virtualplant =
kernel: [<ffffffff8000ac12>] =
kmem_cache_alloc+0x6c/0x76
Feb 24 15:09:51 virtualplant kernel: =
[<ffffffff80012658>] getname+0x25/0x1c2
Feb 24 15:09:51 =
virtualplant kernel: [<ffffffff80019cba>] =
do_sys_open+0x17/0xbe
Feb 24 15:09:51 virtualplant kernel: =
[<ffffffff8005d28d>] =
tracesys+0xd5/0xe0
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">Then I need to cycle the box's =
power.
size=3D"3">
12px;">I'm implementing a multi-layer defense against =
this.
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">1) Try to prevent input that might blow up a =
process. However, this will be imperfect.
align=3D"">
class=3D"Apple-style-span" style=3D"font-size: 12px;">2) Kill apache =
httpd processes occasionally, to control the effect of slow perl memory =
leaks. I'll do this by setting
href=3D"http://httpd.apache.org/docs/2.0/mod/worker.html">
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">
color=3D"#000000">MPM Worker
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">
href=3D"http://httpd.apache.org/docs/2.0/mod/mpm_common.html #maxrequestspe=
rchild">
class=3D"Apple-style-span" style=3D"font-size: 12px;">
class=3D"Apple-style-span" =
color=3D"#000000">MaxRequestsPerChild
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;"> to some modest value. (I'll try =
100.)
size=3D"3">
12px;">3) Kill processes that grow too big, which concerns this =
message.
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">In bash, ulimit sets user resource limits. =
With mod_perl on Apache
href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod">
class=3D"Apple-style-span" style=3D"font-size: 12px;">
class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;"> controls the size of httpd processes. =
Both eventually call setrlimit(int resource, const struct rlimit =
*rlim).
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">With
href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod">
class=3D"Apple-style-span" style=3D"font-size: 12px;">
class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;"> one can put this in the =
httpd.conf:
class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: 12px;">
type=3D"cite">
medium; white-space: normal; ">
face=3D"Helvetica" size=3D"3">
style=3D"font-size: 12px; ">PerlModule Apache2::Resource
# set child memory limit in megabytes
# RLIMIT_AS (address space) will work to limit the size of a process on =
Linux
PerlSetEnv PERL_RLIMIT_AS 1000:1100
class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: 12px; "># this loads =
Apache2::Resource for each new httpd; that will set the ulimits from the =
Perl environment variables
PerlChildInitHandler Apache2::Resource =
align=3D"">
class=3D"Apple-style-span" style=3D"font-size: 12px;">OK, that kills big =
processes. What happens next is that Perl runs out of memory =
(outputs "Out of Memory!") and calls the __DIE__ signal handler. So, my =
plan is to catch the signal, redirect the browser to an error page, and =
finally kill the process. Before the http Request handler is called I =
say:
align=3D"">$SIG{__DIE__} =3D =
\&oversizedHandler;
Then =
when __DIE__ fires the code below runs.
type=3D"cite">use CGI;
use English;
use BSD::Resource =
qw(setrlimit getrlimit get_rlimits getrusage);
# SIG handler =
called when __DIE__ fires
sub =
oversizedHandler{
my $a =3D shift;
=
chomp $a;
print STDERR "handler in process =
$PID called with '$a'\n";
# up the soft AS =
limit to the hard limit, so that we've some RAM to use; in this example =
we free up 100 MB, much more than =
needed
my $success =3D =
setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 * 1024 * =
1024);
if( $success ) {
=
print STDERR "set limits to 512*1024 * =
1024\n";
}
=
$cgi =3D CGI->new;
=
print $cgi->redirect( -location =3D> '
href=3D"http://website.com/program.cgi¶m1=3Dvalue1&a mp;param2=3Dva=
lue2)">http://website.com/program.cgi¶m1=3Dvalue1&am p;param2=3Dval=
ue2);
=
CORE::exit();
}
class=3D"Apple-style-span" style=3D"background-color: transparent; =
">
class=3D"Apple-style-span" style=3D"font-size: 12px;">Here's the =
problem. Nothing goes to STDOUT, so I cannot write to the =
browser.
class=3D"Apple-style-span" style=3D"background-color: =
transparent;">
class=3D"Apple-style-span" style=3D"font-size: 12px;">Thus, my question =
is: how can one kill an oversized process and provide feedback to the =
user at the browser?
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">One alternative seems to be to use the =
special variable $^M (
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml">
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">
color=3D"#000000">see the
href=3D"http://perldoc.perl.org/perlvar.html">
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">
color=3D"#000000">perlvar
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;"> manpage for more =
details
class=3D"Apple-style-span" style=3D"font-size: 12px;">) as recommended =
by the
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml">
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">
color=3D"#000000">modperlbook
class=3D"Apple-style-span" size=3D"3">
style=3D"font-size: 12px;">. How does one determine =
whether -DPERL_EMERGENCY_SBRK is defined, and if one does that does =
STDOUT still work?
class=3D"Apple-style-span" color=3D"#00000000">
class=3D"Apple-style-span" style=3D"background-color: transparent; =
">BR
color=3D"#00000000">
style=3D"background-color: =
transparent;">A
apple-content-edited=3D"true">Arthur P. =
Goldberg
Research =
Scientist in Bioinformatics
Plant Systems Biology =
Laboratory
href=3D"http://www.virtualplant.org">www.virtualplant.org
align=3D"">
Visiting Academic
align=3D"">Computer Science Department
Courant =
Institute of Mathematical Sciences
href=3D"http://www.cs.nyu.edu/artg">www.cs.nyu.edu/artg
align=3D"">
href=3D"mailto:artg@cs.nyu.edu">artg@cs.nyu.edu
align=3D"">New York University212 =
995-4918
Coruzzi Lab
8th Floor =
Silver Building
1009 Silver =
Center
100 Washington Sq East
align=3D"">New York NY =
10003-6688
=
--Apple-Mail-1-126206485--
Re: Killing oversized Perl processes
am 11.03.2010 23:04:21 von Perrin Harkins
On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG wrote:
> 2) Kill apache httpd processes occasionally, to control the effect of slo=
w
> perl memory leaks. I'll do this by setting=A0MPM Worker=A0MaxRequestsPerC=
hild=A0to
> some modest value. (I'll try 100.)
You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.
> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process.
A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.
Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.
> How
> does one determine whether=A0-DPERL_EMERGENCY_SBRK is defined, and if one=
does
> that does STDOUT still work?
It's really not much different from what you're already doing, but you
can check your compile options with perl -V.
- Perrin
Fwd: Killing oversized Perl processes
am 12.03.2010 09:11:17 von William T
On Thu, Mar 11, 2010 at 1:41 PM, ARTHUR GOLDBERG wrote:
>
> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process. Before the http Request handler is called I say=
:
Too complicated. =A0Use Apache2::SizeLimit. =A0It runs as a cleanup
handler which means your user will have gotten served, and then the
child will die. =A0You can also write your own cleanup handler to
monitor sizes and if you want without having to do anything like
catching a __DIE__.
> Thus, my question is: how can one kill an oversized process and provide
> feedback to the user at the browser?
In general kill during cleanup phase. =A0Apache2::SizeLimit does just that.
I would also argue that you should probably fix your memory leaks.
Sometimes all it takes is a few days of analysis. =A0That's what
happened with me.
You can also segment out functionality to different Apache servers so
that every Apache/mod_perl child does not have to pull in the entire
code base for the website.
-wjt
Re: Killing oversized Perl processes
am 12.03.2010 10:11:53 von Rolf Schaufelberger
Am 11.03.2010 um 22:41 schrieb ARTHUR GOLDBERG:
> Running Perl programs in mod_perl in Apache (2.2) on RHEL:
>=20
>> [10 artg@virtualplant:/etc]$ cat /etc/redhat-release
>> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
>> [11 artg@virtualplant:/etc]$ uname -r
>> 2.6.18-164.11.1.el5
> Occasionally a process grows so large that it freezes the system:
I had a similar problem, ending with the kernel message:
no more processes to kill, giving up.
so I installed=20
Apache2::Resource
and set
# set limit to 500MB
PerlSetEnv PERL_RLIMIT_AS 500
PerlChildInitHandler Apache2::Resource
and now the problem dossn't appear again.
And I don't think that Apache2::SizeLimit=20
can handle this issue, since it checks the size after the request, which =
may help if you have a memory leak,=20
but my problem was caused by a single request and the process grew very =
fast during this request, eating up all memory.=20
Setting RLIMIT is handled by the OS , so that process get's killed when =
it grows to large.
>=20
>> several of them will use so much memory that kswapd takes all the =
CPU:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND=20
>> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1=20
>> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0=20
> and
>=20
>=20
>=20
>> from /etc/httpd/logs/error_log
>> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing =
the http daemon from connecting to network port 3306 For complete =
SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
>> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query =
packet.
>> Feb 24 14:55:06 virtualplant last message repeated 6 times
>> Feb 24 15:00:44 virtualplant last message repeated 3 times
>> Feb 24 15:00:55 virtualplant last message repeated 5 times
>> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to =
128.122.128.24 port 67
>> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0
>> Feb 24 15:09:51 virtualplant kernel:=20
>> Feb 24 15:09:51 virtualplant kernel: Call Trace:
>> Feb 24 15:09:51 virtualplant kernel: [] =
out_of_memory+0x8e/0x2f3
>> Feb 24 15:09:51 virtualplant kernel: [] =
__alloc_pages+0x245/0x2ce
>> Feb 24 15:09:51 virtualplant kernel: [] =
cache_grow+0x133/0x3c1
>> Feb 24 15:09:51 virtualplant kernel: [] =
cache_alloc_refill+0x136/0x186
>> Feb 24 15:09:51 virtualplant kernel: [] =
kmem_cache_alloc+0x6c/0x76
>> Feb 24 15:09:51 virtualplant kernel: [] =
getname+0x25/0x1c2
>> Feb 24 15:09:51 virtualplant kernel: [] =
do_sys_open+0x17/0xbe
>> Feb 24 15:09:51 virtualplant kernel: [] =
tracesys+0xd5/0xe0
>=20
> Then I need to cycle the box's power.
>=20
> I'm implementing a multi-layer defense against this.=20
>=20
> 1) Try to prevent input that might blow up a process. However, this =
will be imperfect.
>=20
> 2) Kill apache httpd processes occasionally, to control the effect of =
slow perl memory leaks. I'll do this by setting MPM Worker =
MaxRequestsPerChild to some modest value. (I'll try 100.)
>=20
> 3) Kill processes that grow too big, which concerns this message.=20
>=20
> In bash, ulimit sets user resource limits. With mod_perl on Apache =
Apache2::Resource controls the size of httpd processes. Both eventually =
call setrlimit(int resource, const struct rlimit *rlim).=20
>=20
> With Apache2::Resource one can put this in the httpd.conf:
>=20
>=20
>> PerlModule Apache2::Resource
>> # set child memory limit in megabytes
>> # RLIMIT_AS (address space) will work to limit the size of a process =
on Linux
>> PerlSetEnv PERL_RLIMIT_AS 1000:1100
>>=20
>> # this loads Apache2::Resource for each new httpd; that will set the =
ulimits from the Perl environment variables
>> PerlChildInitHandler Apache2::Resource=20
>>=20
>=20
> OK, that kills big processes. What happens next is that Perl runs out =
of memory (outputs "Out of Memory!") and calls the __DIE__ signal =
handler. So, my plan is to catch the signal, redirect the browser to an =
error page, and finally kill the process. Before the http Request =
handler is called I say:
>=20
>> $SIG{__DIE__} =3D \&oversizedHandler;
>>=20
> Then when __DIE__ fires the code below runs.
>=20
>> use CGI;=20
>> use English;
>> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>>=20
>> # SIG handler called when __DIE__ fires
>> sub oversizedHandler{
>> my $a =3D shift;
>> chomp $a;
>> print STDERR "handler in process $PID called with '$a'\n";
>>=20
>> # up the soft AS limit to the hard limit, so that we've some RAM =
to use; in this example we free up 100 MB, much more than needed
>> my $success =3D setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 =
* 1024 * 1024);
>> if( $success ) {
>> print STDERR "set limits to 512*1024 * 1024\n";
>> }
>>=20
>> $cgi =3D CGI->new;
>> print $cgi->redirect( -location =3D> =
'http://website.com/program.cgi¶m1=3Dvalue1¶m2=3Dval ue2);
>>=20
>> CORE::exit();
>> }
>>=20
> Here's the problem. Nothing goes to STDOUT, so I cannot write to the =
browser.=20
>=20
> Thus, my question is: how can one kill an oversized process and =
provide feedback to the user at the browser?
>=20
> One alternative seems to be to use the special variable $^M (see the =
perlvar manpage for more details) as recommended by the modperlbook. How =
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one =
does that does STDOUT still work?
>=20
> BR
>=20
> A
>=20
> Arthur P. Goldberg
>=20
> Research Scientist in Bioinformatics
> Plant Systems Biology Laboratory
> www.virtualplant.org
>=20
> Visiting Academic
> Computer Science Department
> Courant Institute of Mathematical Sciences
> www.cs.nyu.edu/artg
>=20
> artg@cs.nyu.edu
> New York University
> 212 995-4918
> Coruzzi Lab
> 8th Floor Silver Building
> 1009 Silver Center
> 100 Washington Sq East
> New York NY 10003-6688
>=20
>=20
>=20
Mit freundlichen Grüßen
Rolf Schaufelberger
Geschäftsführer=20
plusW GmbH
Vorstadtstr. 61 -67 Tel. 07181 47 47 305
73614 Schorndorf Fax. 07181 47 45 344
www.plusw.de
www.mypixler.com
www.calendrino.de
Re: Killing oversized Perl processes
am 14.03.2010 16:22:43 von ArthurG
This is a multi-part message in MIME format.
--------------040807000909090007080907
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Hi Perrin
Thanks all for your responses. I'd like to try your suggestion, but
cannot find documentation on "apache error page handlers". Presumably
that's code that Apache runs if an httpd dies.
mod_perl: HTTP
/Handlers
describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors./
Perhaps you're referring to Apache's ErrorDocument
(also
described in Custom Error Response
). While that can
run local Perl (in a new process of course) as in the example
| | ErrorDocument 500 /cgi-bin/crash-recover.pl||
it isn't clear to me how to access $r (which I assume means the Request).
However, I could write a little info to a file which that
crash-recover.pl reads. It will only run occasionally.
And given your observation
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in crash-recover.pl. Alternatively, I could
write a handler that did
$original_handler = $SIG{__DIE__};
sub myHandler {
if( out_of_memory ) {
# increase RLIMIT_AS a little so memory can be allocated
# write some info (especially user identity) to a file, that
crash-recover.pl will use
} else {
$original_handler( @_ );
}
}
BR
A
Perrin Harkins wrote:
> On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG wrote:
>
>> 2) Kill apache httpd processes occasionally, to control the effect of slow
>> perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
>> some modest value. (I'll try 100.)
>>
>
> You definitely should be doing that, and possibly running
> Apache2::SizeLimit as well. There's plenty of documentation about
> this on the site.
>
>
>> OK, that kills big processes. What happens next is that Perl runs out of
>> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
>> my plan is to catch the signal, redirect the browser to an error page, and
>> finally kill the process.
>>
>
> A simpler solution would be to set something in $r->notes from your
> DIE handler and use that in an apache error page handler to determine
> what to show.
>
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
>
>> How
>> does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
>> that does STDOUT still work?
>>
>
> It's really not much different from what you're already doing, but you
> can check your compile options with perl -V.
>
> - Perrin
>
>
--
Arthur P. Goldberg, PhD
Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org
Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg
artg@cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building
--------------040807000909090007080907
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Hi Perrin
Thanks all for your responses. I'd like to try
your suggestion, but cannot find documentation on "apache error page
handlers". Presumably that's code that Apache runs if an httpd dies.
style="font-size: 14px; font-weight: bold;">
href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">mod_perl:
HTTP
href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">Handlers
class="Apple-style-span"
style="font-size: medium; font-style: normal; font-weight: normal;"> describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors.
Perhaps you're referring to Apache's
href="http://httpd.apache.org/docs/2.0/mod/core.html#errordo cument">ErrorDocument
(also described in
href="http://httpd.apache.org/docs/2.0/custom-error.html">Cu stom Error
Response). While that can run local Perl (in a new process of
course) as in the example
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">ErrorDocument
500 /cgi-bin/crash-recover.pl
it isn't clear to me how to access $r (which I assume means the
Request).
However, I could write a little info to a file which that
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl
reads. It will only run occasionally.
And given your observation
Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.
it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl.
Alternatively, I could write a handler that did
$original_handler = $SIG{__DIE__};
sub myHandler {
if( out_of_memory ) {
# increase RLIMIT_AS a little so memory can be allocated
# write some info (especially user identity) to a file, that
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl
will use
} else {
$original_handler( @_ );
}
}
BR
class="Apple-style-span" style="white-space: normal;">A
Perrin Harkins wrote:
cite="mid:66887a3d1003111404m2127564dq1ce498ddd769498@mail.g mail.com"
type="cite">
On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG wrote:
2) Kill apache httpd processes occasionally, to control the effect of slow
perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
some modest value. (I'll try 100.)
You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.
OK, that kills big processes. What happens next is that Perl runs out of
memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
my plan is to catch the signal, redirect the browser to an error page, and
finally kill the process.
A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.
Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.
How
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
that does STDOUT still work?
It's really not much different from what you're already doing, but you
can check your compile options with perl -V.
- Perrin
--
Arthur P. Goldberg, PhD
Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences
New York University
100 Washington Sq East
8th Floor Silver Building
--------------040807000909090007080907--
Re: Killing oversized Perl processes
am 15.03.2010 23:55:16 von Perrin Harkins
On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg wrote:
> Perhaps you're referring to Apache's=A0ErrorDocument (also described in C=
ustom
> Error Response).
Yes, that's what I meant.
> While that can run local Perl (in a new process of course)
> as in the example
> =A0 =A0 ErrorDocument 500 /cgi-bin/crash-recover.pl
> it isn't clear to me how to access $r (which I assume means the Request).
You can send it to any URL you like, including one handled by mod_perl.
- Perrin
Re: Killing oversized Perl processes
am 16.03.2010 01:22:29 von ArthurG
This is a multi-part message in MIME format.
--------------000202040908030509030209
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Thanks Perrin
Unfortunately, this doesn't work.
First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache
ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.
Second, if Apache detects a server error and redirects to a script, that
script runs in a new process and has no access to the request object $r.
It can access the REDIRECT environment variables as discussed in Custom
Error Response .
At this point I think that the best thing to do is use
MaxRequestsPerChild
and Apache2::SizeLimit
to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.
BR
A
Perrin Harkins wrote:
> On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg wrote:
>
>> Perhaps you're referring to Apache's ErrorDocument (also described in Custom
>> Error Response).
>>
>
> Yes, that's what I meant.
>
>
>> While that can run local Perl (in a new process of course)
>> as in the example
>> ErrorDocument 500 /cgi-bin/crash-recover.pl
>> it isn't clear to me how to access $r (which I assume means the Request).
>>
>
> You can send it to any URL you like, including one handled by mod_perl.
>
> - Perrin
>
>
--
Arthur P. Goldberg, PhD
Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org
Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg
artg@cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building
--------------000202040908030509030209
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Thanks Perrin
Unfortunately, this doesn't work.
First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache
ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.
Second, if Apache detects a server error and redirects to a script,
that script runs in a new process and has no access to the request
object $r. It can access the REDIRECT environment variables as
discussed in
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;">
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;">
href="http://httpd.apache.org/docs/2.0/custom-error.html"
style="color: rgb(85, 26, 139);">Custom Error Response
..
At this point I think that the best thing to do is use
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;">
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;">
class="Apple-converted-space">
href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#m axrequestsperchild"
id="hmps" title="MaxRequestsPerChild">MaxRequestsPerChild
and
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;">
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;">
href="http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ap i/Apache2/SizeLimit.pod"
id="j:1-" title="Apache2::SizeLimit">Apache2::SizeLimit
to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.
BR
A
Perrin Harkins wrote:
cite="mid:66887a3d1003151555i4704ea31w8100fbb20042e266@mail. gmail.com"
type="cite">
On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg wrote:
Perhaps you're referring to Apache's ErrorDocument (also described in Custom
Error Response).
Yes, that's what I meant.
While that can run local Perl (in a new process of course)
as in the example
ErrorDocument 500 /cgi-bin/crash-recover.pl
it isn't clear to me how to access $r (which I assume means the Request).
You can send it to any URL you like, including one handled by mod_perl.
- Perrin
--
Arthur P. Goldberg, PhD
Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences
New York University
100 Washington Sq East
8th Floor Silver Building
--------------000202040908030509030209--