Killing oversized Perl processes

Killing oversized Perl processes

am 11.03.2010 22:41:38 von ArthurG

--Apple-Mail-1-126206485
Content-Type: text/plain;
charset=US-ASCII;
format=flowed;
delsp=yes
Content-Transfer-Encoding: 7bit

Running Perl programs in mod_perl in Apache (2.2) on RHEL:

> [10 artg@virtualplant:/etc]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
> [11 artg@virtualplant:/etc]$ uname -r
> 2.6.18-164.11.1.el5

Occasionally a process grows so large that it freezes the system:

> several of them will use so much memory that kswapd takes all the CPU:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
and


> from /etc/httpd/logs/error_log
> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing
> the http daemon from connecting to network port 3306 For complete
> SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.
> Feb 24 14:55:06 virtualplant last message repeated 6 times
> Feb 24 15:00:44 virtualplant last message repeated 3 times
> Feb 24 15:00:55 virtualplant last message repeated 5 times
> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to
> 128.122.128.24 port 67
> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked oom-
> killer: gfp_mask=0xd0, order=0, oomkilladj=0
> Feb 24 15:09:51 virtualplant kernel:
> Feb 24 15:09:51 virtualplant kernel: Call Trace:
> Feb 24 15:09:51 virtualplant kernel: []
> out_of_memory+0x8e/0x2f3
> Feb 24 15:09:51 virtualplant kernel: []
> __alloc_pages+0x245/0x2ce
> Feb 24 15:09:51 virtualplant kernel: [] cache_grow
> +0x133/0x3c1
> Feb 24 15:09:51 virtualplant kernel: []
> cache_alloc_refill+0x136/0x186
> Feb 24 15:09:51 virtualplant kernel: []
> kmem_cache_alloc+0x6c/0x76
> Feb 24 15:09:51 virtualplant kernel: [] getname
> +0x25/0x1c2
> Feb 24 15:09:51 virtualplant kernel: []
> do_sys_open+0x17/0xbe
> Feb 24 15:09:51 virtualplant kernel: [] tracesys
> +0xd5/0xe0

Then I need to cycle the box's power.

I'm implementing a multi-layer defense against this.

1) Try to prevent input that might blow up a process. However, this
will be imperfect.

2) Kill apache httpd processes occasionally, to control the effect of
slow perl memory leaks. I'll do this by setting MPM Worker
MaxRequestsPerChild to some modest value. (I'll try 100.)

3) Kill processes that grow too big, which concerns this message.

In bash, ulimit sets user resource limits. With mod_perl on Apache
Apache2::Resource controls the size of httpd processes. Both
eventually call setrlimit(int resource, const struct rlimit *rlim).

With Apache2::Resource one can put this in the httpd.conf:

> PerlModule Apache2::Resource
> # set child memory limit in megabytes
> # RLIMIT_AS (address space) will work to limit the size of a process
> on Linux
> PerlSetEnv PERL_RLIMIT_AS 1000:1100
> # this loads Apache2::Resource for each new httpd; that will set the
> ulimits from the Perl environment variables
> PerlChildInitHandler Apache2::Resource

OK, that kills big processes. What happens next is that Perl runs out
of memory (outputs "Out of Memory!") and calls the __DIE__ signal
handler. So, my plan is to catch the signal, redirect the browser to
an error page, and finally kill the process. Before the http Request
handler is called I say:

> $SIG{__DIE__} = \&oversizedHandler;
>
Then when __DIE__ fires the code below runs.

> use CGI;
> use English;
> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>
> # SIG handler called when __DIE__ fires
> sub oversizedHandler{
> my $a = shift;
> chomp $a;
> print STDERR "handler in process $PID called with '$a'\n";
>
> # up the soft AS limit to the hard limit, so that we've some RAM
> to use; in this example we free up 100 MB, much more than needed
> my $success = setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100
> * 1024 * 1024);
> if( $success ) {
> print STDERR "set limits to 512*1024 * 1024\n";
> }
>
> $cgi = CGI->new;
> print $cgi->redirect( -location => 'http://website.com/program.cgi¶m1=value1¶m2=value2)
> ;
>
> CORE::exit();
> }
>
Here's the problem. Nothing goes to STDOUT, so I cannot write to the
browser.

Thus, my question is: how can one kill an oversized process and
provide feedback to the user at the browser?

One alternative seems to be to use the special variable $^M (see the
perlvar manpage for more details) as recommended by the modperlbook.
How does one determine whether -DPERL_EMERGENCY_SBRK is defined, and
if one does that does STDOUT still work?

BR

A

Arthur P. Goldberg

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences
www.cs.nyu.edu/artg

artg@cs.nyu.edu
New York University
212 995-4918
Coruzzi Lab
8th Floor Silver Building
1009 Silver Center
100 Washington Sq East
New York NY 10003-6688




--Apple-Mail-1-126206485
Content-Type: text/html;
charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

-webkit-line-break: after-white-space; ">

Running Perl programs =
in mod_perl in Apache (2.2) on =
RHEL:

class=3D"Apple-style-span" color=3D"#000000">[10 =
artg@virtualplant:/etc]$ cat /etc/redhat-release
Red Hat Enterprise =
Linux Server release 5.4 (Tikanga)
[11 artg@virtualplant:/etc]$ uname =
-r
2.6.18-164.11.1.el5

v>

Occasionally a process grows so large that it freezes =
the system:

class=3D"Apple-style-span" color=3D"#000000">several of them will use so =
much memory that kswapd takes all the CPU:
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3"> class=3D"Apple-style-span" style=3D"font-size: 12px;">   PID =
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  349 =
root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
  348 root 20 =
-5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3"> class=3D"Apple-style-span" style=3D"font-size: =
12px;"> 

and

align=3D"">

color=3D"#000000">from /etc/httpd/logs/error_log
Feb
24 14:35:32 virtualplant setroubleshoot: SELinux is preventing the http
daemon from connecting to network port 3306 For complete SELinux
messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
Feb 24 =
14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.
Feb =
24 14:55:06 virtualplant last message repeated 6 times
Feb 24 =
15:00:44 virtualplant last message repeated 3 times
Feb 24 15:00:55 =
virtualplant last message repeated 5 times
Feb 24 15:01:21 =
virtualplant dhclient: DHCPREQUEST on eth0 to 128.122.128.24 port =
67
Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0
Feb 24 =
15:09:51 virtualplant kernel:
Feb 24 15:09:51 virtualplant kernel: =
Call Trace:
Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff800c6076>] out_of_memory+0x8e/0x2f3
Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff8000f487>] =
__alloc_pages+0x245/0x2ce
Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff80017812>] cache_grow+0x133/0x3c1
Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff8005c2e5>] =
cache_alloc_refill+0x136/0x186
Feb 24 15:09:51 virtualplant =
kernel:  [<ffffffff8000ac12>] =
kmem_cache_alloc+0x6c/0x76
Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff80012658>] getname+0x25/0x1c2
Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff80019cba>] =
do_sys_open+0x17/0xbe
Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff8005d28d>] =
tracesys+0xd5/0xe0

class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">Then I need to cycle the box's =
power.

size=3D"3"> 12px;">I'm implementing a multi-layer defense against =
this. 

class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">1) Try to prevent input that might blow up a =
process. However, this will be imperfect.

align=3D""> class=3D"Apple-style-span" style=3D"font-size: 12px;">2) Kill apache =
httpd processes occasionally, to control the effect of slow perl memory =
leaks. I'll do this by setting 
href=3D"http://httpd.apache.org/docs/2.0/mod/worker.html"> class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> color=3D"#000000">MPM Worker class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">  href=3D"http://httpd.apache.org/docs/2.0/mod/mpm_common.html #maxrequestspe=
rchild"> class=3D"Apple-style-span" style=3D"font-size: 12px;"> class=3D"Apple-style-span" =
color=3D"#000000">MaxRequestsPerChild
class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> to some modest value. (I'll try =
100.)

size=3D"3"> 12px;">3) Kill processes that grow too big, which concerns this =
message. 

class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">In bash, ulimit sets user resource limits. =
With mod_perl on Apache 
href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod"> class=3D"Apple-style-span" style=3D"font-size: 12px;"> class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource
class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> controls the size of httpd processes. =
Both eventually call setrlimit(int resource, const struct rlimit =
*rlim). 

class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">With   href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod"> class=3D"Apple-style-span" style=3D"font-size: 12px;"> class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource
class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> one can put this in the =
httpd.conf:


class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: 12px;">
type=3D"cite"> medium; white-space: normal; ">

face=3D"Helvetica" size=3D"3">
style=3D"font-size: 12px; ">PerlModule Apache2::Resource
# set child memory limit in megabytes
# RLIMIT_AS (address space) will work to limit the size of a process on =
Linux
PerlSetEnv PERL_RLIMIT_AS 1000:1100

class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3">
class=3D"Apple-style-span" style=3D"font-size: 12px; "># this loads =
Apache2::Resource for each new httpd; that will set the ulimits from the =
Perl environment variables
PerlChildInitHandler Apache2::Resource =

align=3D""> class=3D"Apple-style-span" style=3D"font-size: 12px;">OK, that kills big =
processes. What happens next is that Perl runs out of memory =
(outputs "Out of Memory!") and calls the __DIE__ signal handler. So, my =
plan is to catch the signal, redirect the browser to an error page, and =
finally kill the process. Before the http Request handler is called I =
say:

align=3D"">$SIG{__DIE__} =3D =
\&oversizedHandler;

Then =
when __DIE__ fires the code below runs.

type=3D"cite">use CGI; 
use English;
use BSD::Resource =
qw(setrlimit getrlimit get_rlimits getrusage);

# SIG handler =
called when __DIE__ fires

sub =
oversizedHandler{
    my $a =3D shift;
   =
 chomp $a;
    print STDERR "handler in process =
$PID called with '$a'\n";

    # up the soft AS =
limit to the hard limit, so that we've some RAM to use; in this example =
we free up 100 MB, much more than =
needed
        my $success =3D =
setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 * 1024 * =
1024);
    if( $success ) {
   =
     print STDERR "set limits to 512*1024 * =
1024\n";
    }

   =
 $cgi =3D CGI->new;
   =
 print $cgi->redirect( -location =3D> ' href=3D"http://website.com/program.cgi&param1=3Dvalue1&a mp;param2=3Dva=
lue2)">http://website.com/program.cgi&param1=3Dvalue1&am p;param2=3Dval=
ue2)
;

   =
 CORE::exit();
}

class=3D"Apple-style-span" style=3D"background-color: transparent; =
"> class=3D"Apple-style-span" style=3D"font-size: 12px;">Here's the =
problem. Nothing goes to STDOUT, so I cannot write to the =
browser. 

class=3D"Apple-style-span" style=3D"background-color: =
transparent;"> class=3D"Apple-style-span" style=3D"font-size: 12px;">Thus, my question =
is: how can one kill an oversized process and provide feedback to the =
user at the browser?

class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">One alternative seems to be to use the =
special variable $^M (
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml"> class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> color=3D"#000000">see the  href=3D"http://perldoc.perl.org/perlvar.html"> class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> color=3D"#000000">perlvar class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> manpage for more =
details
class=3D"Apple-style-span" style=3D"font-size: 12px;">) as recommended =
by the 
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml"> class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;"> color=3D"#000000">modperlbook class=3D"Apple-style-span" size=3D"3"> style=3D"font-size: 12px;">. How does one determine =
whether -DPERL_EMERGENCY_SBRK is defined, and if one does that does =
STDOUT still work?

class=3D"Apple-style-span" color=3D"#00000000"> class=3D"Apple-style-span" style=3D"background-color: transparent; =
">BR

color=3D"#00000000"> style=3D"background-color: =
transparent;">A

apple-content-edited=3D"true">
Arthur P. =
Goldberg

Research =
Scientist in Bioinformatics
Plant Systems Biology =
Laboratory
align=3D"">
Visiting Academic
align=3D"">Computer Science Department
Courant =
Institute of Mathematical Sciences
align=3D"">
align=3D"">New York University
212 =
995-4918
Coruzzi Lab
8th Floor =
Silver Building
1009 Silver =
Center
100 Washington Sq East
align=3D"">New York NY =
10003-6688



=

--Apple-Mail-1-126206485--

Re: Killing oversized Perl processes

am 11.03.2010 23:04:21 von Perrin Harkins

On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG wrote:
> 2) Kill apache httpd processes occasionally, to control the effect of slo=
w
> perl memory leaks. I'll do this by setting=A0MPM Worker=A0MaxRequestsPerC=
hild=A0to
> some modest value. (I'll try 100.)

You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.

> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process.

A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.

Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.

> How
> does one determine whether=A0-DPERL_EMERGENCY_SBRK is defined, and if one=
does
> that does STDOUT still work?

It's really not much different from what you're already doing, but you
can check your compile options with perl -V.

- Perrin

Fwd: Killing oversized Perl processes

am 12.03.2010 09:11:17 von William T

On Thu, Mar 11, 2010 at 1:41 PM, ARTHUR GOLDBERG wrote:
>
> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process. Before the http Request handler is called I say=
:

Too complicated. =A0Use Apache2::SizeLimit. =A0It runs as a cleanup
handler which means your user will have gotten served, and then the
child will die. =A0You can also write your own cleanup handler to
monitor sizes and if you want without having to do anything like
catching a __DIE__.

> Thus, my question is: how can one kill an oversized process and provide
> feedback to the user at the browser?

In general kill during cleanup phase. =A0Apache2::SizeLimit does just that.

I would also argue that you should probably fix your memory leaks.
Sometimes all it takes is a few days of analysis. =A0That's what
happened with me.

You can also segment out functionality to different Apache servers so
that every Apache/mod_perl child does not have to pull in the entire
code base for the website.

-wjt

Re: Killing oversized Perl processes

am 12.03.2010 10:11:53 von Rolf Schaufelberger

Am 11.03.2010 um 22:41 schrieb ARTHUR GOLDBERG:

> Running Perl programs in mod_perl in Apache (2.2) on RHEL:
>=20
>> [10 artg@virtualplant:/etc]$ cat /etc/redhat-release
>> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
>> [11 artg@virtualplant:/etc]$ uname -r
>> 2.6.18-164.11.1.el5
> Occasionally a process grows so large that it freezes the system:

I had a similar problem, ending with the kernel message:
no more processes to kill, giving up.

so I installed=20
Apache2::Resource
and set
# set limit to 500MB
PerlSetEnv PERL_RLIMIT_AS 500
PerlChildInitHandler Apache2::Resource

and now the problem dossn't appear again.

And I don't think that Apache2::SizeLimit=20
can handle this issue, since it checks the size after the request, which =
may help if you have a memory leak,=20
but my problem was caused by a single request and the process grew very =
fast during this request, eating up all memory.=20
Setting RLIMIT is handled by the OS , so that process get's killed when =
it grows to large.

>=20
>> several of them will use so much memory that kswapd takes all the =
CPU:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND=20
>> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1=20
>> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0=20
> and
>=20
>=20
>=20
>> from /etc/httpd/logs/error_log
>> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing =
the http daemon from connecting to network port 3306 For complete =
SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
>> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query =
packet.
>> Feb 24 14:55:06 virtualplant last message repeated 6 times
>> Feb 24 15:00:44 virtualplant last message repeated 3 times
>> Feb 24 15:00:55 virtualplant last message repeated 5 times
>> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to =
128.122.128.24 port 67
>> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0
>> Feb 24 15:09:51 virtualplant kernel:=20
>> Feb 24 15:09:51 virtualplant kernel: Call Trace:
>> Feb 24 15:09:51 virtualplant kernel: [] =
out_of_memory+0x8e/0x2f3
>> Feb 24 15:09:51 virtualplant kernel: [] =
__alloc_pages+0x245/0x2ce
>> Feb 24 15:09:51 virtualplant kernel: [] =
cache_grow+0x133/0x3c1
>> Feb 24 15:09:51 virtualplant kernel: [] =
cache_alloc_refill+0x136/0x186
>> Feb 24 15:09:51 virtualplant kernel: [] =
kmem_cache_alloc+0x6c/0x76
>> Feb 24 15:09:51 virtualplant kernel: [] =
getname+0x25/0x1c2
>> Feb 24 15:09:51 virtualplant kernel: [] =
do_sys_open+0x17/0xbe
>> Feb 24 15:09:51 virtualplant kernel: [] =
tracesys+0xd5/0xe0
>=20
> Then I need to cycle the box's power.
>=20
> I'm implementing a multi-layer defense against this.=20
>=20
> 1) Try to prevent input that might blow up a process. However, this =
will be imperfect.
>=20
> 2) Kill apache httpd processes occasionally, to control the effect of =
slow perl memory leaks. I'll do this by setting MPM Worker =
MaxRequestsPerChild to some modest value. (I'll try 100.)
>=20
> 3) Kill processes that grow too big, which concerns this message.=20
>=20
> In bash, ulimit sets user resource limits. With mod_perl on Apache =
Apache2::Resource controls the size of httpd processes. Both eventually =
call setrlimit(int resource, const struct rlimit *rlim).=20
>=20
> With Apache2::Resource one can put this in the httpd.conf:
>=20
>=20
>> PerlModule Apache2::Resource
>> # set child memory limit in megabytes
>> # RLIMIT_AS (address space) will work to limit the size of a process =
on Linux
>> PerlSetEnv PERL_RLIMIT_AS 1000:1100
>>=20
>> # this loads Apache2::Resource for each new httpd; that will set the =
ulimits from the Perl environment variables
>> PerlChildInitHandler Apache2::Resource=20
>>=20
>=20
> OK, that kills big processes. What happens next is that Perl runs out =
of memory (outputs "Out of Memory!") and calls the __DIE__ signal =
handler. So, my plan is to catch the signal, redirect the browser to an =
error page, and finally kill the process. Before the http Request =
handler is called I say:
>=20
>> $SIG{__DIE__} =3D \&oversizedHandler;
>>=20
> Then when __DIE__ fires the code below runs.
>=20
>> use CGI;=20
>> use English;
>> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>>=20
>> # SIG handler called when __DIE__ fires
>> sub oversizedHandler{
>> my $a =3D shift;
>> chomp $a;
>> print STDERR "handler in process $PID called with '$a'\n";
>>=20
>> # up the soft AS limit to the hard limit, so that we've some RAM =
to use; in this example we free up 100 MB, much more than needed
>> my $success =3D setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 =
* 1024 * 1024);
>> if( $success ) {
>> print STDERR "set limits to 512*1024 * 1024\n";
>> }
>>=20
>> $cgi =3D CGI->new;
>> print $cgi->redirect( -location =3D> =
'http://website.com/program.cgi¶m1=3Dvalue1¶m2=3Dval ue2);
>>=20
>> CORE::exit();
>> }
>>=20
> Here's the problem. Nothing goes to STDOUT, so I cannot write to the =
browser.=20
>=20
> Thus, my question is: how can one kill an oversized process and =
provide feedback to the user at the browser?
>=20
> One alternative seems to be to use the special variable $^M (see the =
perlvar manpage for more details) as recommended by the modperlbook. How =
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one =
does that does STDOUT still work?
>=20
> BR
>=20
> A
>=20
> Arthur P. Goldberg
>=20
> Research Scientist in Bioinformatics
> Plant Systems Biology Laboratory
> www.virtualplant.org
>=20
> Visiting Academic
> Computer Science Department
> Courant Institute of Mathematical Sciences
> www.cs.nyu.edu/artg
>=20
> artg@cs.nyu.edu
> New York University
> 212 995-4918
> Coruzzi Lab
> 8th Floor Silver Building
> 1009 Silver Center
> 100 Washington Sq East
> New York NY 10003-6688
>=20
>=20
>=20

Mit freundlichen Grüßen
Rolf Schaufelberger
Geschäftsführer=20

plusW GmbH
Vorstadtstr. 61 -67 Tel. 07181 47 47 305
73614 Schorndorf Fax. 07181 47 45 344

www.plusw.de
www.mypixler.com
www.calendrino.de

Re: Killing oversized Perl processes

am 14.03.2010 16:22:43 von ArthurG

This is a multi-part message in MIME format.
--------------040807000909090007080907
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Perrin

Thanks all for your responses. I'd like to try your suggestion, but
cannot find documentation on "apache error page handlers". Presumably
that's code that Apache runs if an httpd dies.
mod_perl: HTTP
/Handlers
describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors./
Perhaps you're referring to Apache's ErrorDocument
(also
described in Custom Error Response
). While that can
run local Perl (in a new process of course) as in the example
| | ErrorDocument 500 /cgi-bin/crash-recover.pl||
it isn't clear to me how to access $r (which I assume means the Request).
However, I could write a little info to a file which that
crash-recover.pl reads. It will only run occasionally.
And given your observation
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in crash-recover.pl. Alternatively, I could
write a handler that did

$original_handler = $SIG{__DIE__};

sub myHandler {
if( out_of_memory ) {
# increase RLIMIT_AS a little so memory can be allocated
# write some info (especially user identity) to a file, that
crash-recover.pl will use
} else {
$original_handler( @_ );
}
}

BR
A


Perrin Harkins wrote:
> On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG wrote:
>
>> 2) Kill apache httpd processes occasionally, to control the effect of slow
>> perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
>> some modest value. (I'll try 100.)
>>
>
> You definitely should be doing that, and possibly running
> Apache2::SizeLimit as well. There's plenty of documentation about
> this on the site.
>
>
>> OK, that kills big processes. What happens next is that Perl runs out of
>> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
>> my plan is to catch the signal, redirect the browser to an error page, and
>> finally kill the process.
>>
>
> A simpler solution would be to set something in $r->notes from your
> DIE handler and use that in an apache error page handler to determine
> what to show.
>
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
>
>> How
>> does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
>> that does STDOUT still work?
>>
>
> It's really not much different from what you're already doing, but you
> can check your compile options with perl -V.
>
> - Perrin
>
>

--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg

artg@cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building



--------------040807000909090007080907
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit







Hi Perrin




Thanks all for your responses. I'd like to try
your suggestion, but cannot find documentation on "apache error page
handlers". Presumably that's code that Apache runs if an httpd dies.

style="font-size: 14px; font-weight: bold;"> href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">mod_perl:
HTTP href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">Handlers class="Apple-style-span"
style="font-size: medium; font-style: normal; font-weight: normal;"> describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors.

Perhaps you're referring to Apache's  href="http://httpd.apache.org/docs/2.0/mod/core.html#errordo cument">ErrorDocument
(also described in href="http://httpd.apache.org/docs/2.0/custom-error.html">Cu stom Error
Response). While that can run local Perl (in a new process of
course) as in the example

      

style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"> class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">ErrorDocument
500 /cgi-bin/crash-recover.pl

it isn't clear to me how to access $r (which I assume means the
Request).

However, I could write a little info to a file which that class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"> class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl
reads. It will only run occasionally.

And given your observation


Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.


it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"> class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl.
Alternatively, I could write a handler that did



$original_handler = $SIG{__DIE__};



sub myHandler {

if( out_of_memory ) {

   # increase RLIMIT_AS a little so memory can be allocated

   # write some info (especially user identity) to a file, that class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"> class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl
will use

} else {

  $original_handler( @_ );

}

}



BR





class="Apple-style-span" style="white-space: normal;">A









Perrin Harkins wrote:
cite="mid:66887a3d1003111404m2127564dq1ce498ddd769498@mail.g mail.com"
type="cite">
On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG  wrote:


2) Kill apache httpd processes occasionally, to control the effect of slow
perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
some modest value. (I'll try 100.)



You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.



OK, that kills big processes. What happens next is that Perl runs out of
memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
my plan is to catch the signal, redirect the browser to an error page, and
finally kill the process.



A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.

Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.



How
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
that does STDOUT still work?



It's really not much different from what you're already doing, but you
can check your compile options with perl -V.

- Perrin





-- 
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory


Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences


New York University
100 Washington Sq East
8th Floor Silver Building





--------------040807000909090007080907--

Re: Killing oversized Perl processes

am 15.03.2010 23:55:16 von Perrin Harkins

On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg wrote:
> Perhaps you're referring to Apache's=A0ErrorDocument (also described in C=
ustom
> Error Response).

Yes, that's what I meant.

> While that can run local Perl (in a new process of course)
> as in the example
>    =A0 =A0 ErrorDocument 500 /cgi-bin/crash-recover.pl
> it isn't clear to me how to access $r (which I assume means the Request).

You can send it to any URL you like, including one handled by mod_perl.

- Perrin

Re: Killing oversized Perl processes

am 16.03.2010 01:22:29 von ArthurG

This is a multi-part message in MIME format.
--------------000202040908030509030209
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Thanks Perrin

Unfortunately, this doesn't work.
First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache
ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.

Second, if Apache detects a server error and redirects to a script, that
script runs in a new process and has no access to the request object $r.
It can access the REDIRECT environment variables as discussed in Custom
Error Response .

At this point I think that the best thing to do is use
MaxRequestsPerChild

and Apache2::SizeLimit

to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.

BR
A

Perrin Harkins wrote:
> On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg wrote:
>
>> Perhaps you're referring to Apache's ErrorDocument (also described in Custom
>> Error Response).
>>
>
> Yes, that's what I meant.
>
>
>> While that can run local Perl (in a new process of course)
>> as in the example
>> ErrorDocument 500 /cgi-bin/crash-recover.pl
>> it isn't clear to me how to access $r (which I assume means the Request).
>>
>
> You can send it to any URL you like, including one handled by mod_perl.
>
> - Perrin
>
>

--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg

artg@cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building



--------------000202040908030509030209
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit







Thanks Perrin



Unfortunately, this doesn't work.

First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache


ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.



Second, if Apache detects a server error and redirects to a script,
that script runs in a new process and has no access to the request
object $r. It can access the REDIRECT environment variables as
discussed in style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"> class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"> href="http://httpd.apache.org/docs/2.0/custom-error.html"
style="color: rgb(85, 26, 139);">Custom Error Response


..



At this point I think that the best thing to do is use

style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"> class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"> class="Apple-converted-space">  href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#m axrequestsperchild"
id="hmps" title="MaxRequestsPerChild">MaxRequestsPerChild
and

style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"> class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"> href="http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ap i/Apache2/SizeLimit.pod"
id="j:1-" title="Apache2::SizeLimit">Apache2::SizeLimit

to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.



BR

A



Perrin Harkins wrote:
cite="mid:66887a3d1003151555i4704ea31w8100fbb20042e266@mail. gmail.com"
type="cite">

On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg  wrote:


Perhaps you're referring to Apache's ErrorDocument (also described in Custom
Error Response).



Yes, that's what I meant.



While that can run local Perl (in a new process of course)
as in the example
       ErrorDocument 500 /cgi-bin/crash-recover.pl
it isn't clear to me how to access $r (which I assume means the Request).



You can send it to any URL you like, including one handled by mod_perl.

- Perrin





-- 
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory


Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences


New York University
100 Washington Sq East
8th Floor Silver Building





--------------000202040908030509030209--