avoiding child death by size limit

Hi,

I have a problem where a mod_perl handler will allocate a lot of
memory when processing a request, and this causes Apache to kill the
child due to exceeding the configure child size limit.

However, the memory allocated will get freed up or re-used by the next
request - I think the memory is just fragmented enough to be
automatically reclaimed by the memory allocator (I've heard that some
mallocs can return memory to the OS in 1 MB chunks.)

Are there any special techniques people use to avoid this situation?
Does SizeLimit count actual memory used or does it just look at the
process size?

Thanks,
ER

Re: avoiding child death by size limit

On Thu, Dec 10, 2009 at 2:28 PM, E R wrote:
> However, the memory allocated will get freed up or re-used by the next
> request

The memory won't get freed unless you undef all the variables
manually. Perl keeps that memory otherwise, even when they go out of
scope.

If you know it will get reused on the next request, then set your size
limit higher.

If it's only being used for the current request and the odds of reuse
on the following requests is low, the best thing is to kill the
process as SizeLimit is doing.

- Perrin

Re: avoiding child death by size limit

E R wrote:
> Hi,
>
> I have a problem where a mod_perl handler will allocate a lot of
> memory when processing a request, and this causes Apache to kill the
> child due to exceeding the configure child size limit.
>
> However, the memory allocated will get freed up or re-used by the next
> request - I think the memory is just fragmented enough to be
> automatically reclaimed by the memory allocator (I've heard that some
> mallocs can return memory to the OS in 1 MB chunks.)
>
> Are there any special techniques people use to avoid this situation?
> Does SizeLimit count actual memory used or does it just look at the
> process size?
>
This is not a direct answer to your question, and it begs for a more
authoritative answer.

I have had problems similar to yours, which I solved by turning what was
original part of my mod_perl handler (or script), into a separate
process. I know it is not elegant, but it seems to work well in my case.
One of the problems is that, as far as I know, once perl has obtained
some memory from the OS, it will never give it back until perl itself
exits. And since by definition with mod_perl normally the perl
interpreter lives as long as the Apache process it is inside of, that
means almost never.
But if perl runs as an external process for a request, then of course it
exits at the end of it, and the memory is returned.

The exact problem I had, was that some processing I had to do involved
parsing XML, and some XML parsing module in the chain (XML::Twig ?) was
leaking a chunk of memory at each request. I ended up with
multi-megabyte Apache children all the time.
So I off-loaded this parsing into a separate process, which wrote (to
disk) its results in the form of a Storable structure. When the
external process was done, the main mod_perl handler sucked in the data
back from the Storable file and deleted it.
Not elegant, but it's been working flawlessly for several years now.

Re: avoiding child death by size limit

On Thu, Dec 10, 2009 at 11:28 AM, E R wrote:
> Hi,
>
> I have a problem where a mod_perl handler will allocate a lot of
> memory when processing a request, and this causes Apache to kill the
> child due to exceeding the configure child size limit.
>
> Are there any special techniques people use to avoid this situation?
> Does SizeLimit count actual memory used or does it just look at the
> process size?

You can make sure the variables that the memory was malloc'd for have
gone out of scope, and there are not trailing references to them.
Perl will then reuse that memory. Occasionaly Perl will free memory
it's not using, but why, where and when can be hard to determine.
It's not always as clear cut as undef'ing the variable.

In terms of managing memory in your mod_perl process you can use any
of the standard techniques:

* change/fix the code (fix memory leak, Tie structure to filesystem/db, ...)
* recycle your children after N number of request are completed
* offload the memory intensive work into a different process space
(another instance of Apache/mod_perl or even just Perl)
* use a system set resource limit (not so good because it can kill
the proc mid-request)
* Apache2::SizeLimit (kill the proc after it's done serving the
request when it grows too big)

These were off the top of my head. THere may be other techniques I missed.

-wjt

Re: avoiding child death by size limit

E R wrote:
> Hi,
>
> I have a problem where a mod_perl handler will allocate a lot of
> memory when processing a request, and this causes Apache to kill the
> child due to exceeding the configure child size limit.

Chances are that a child does not exceed this memory right away, at the
first request. More likely, it uses more and more memory at each
request it processes, and finally after a number of requests, it exceeds
the maximum memory and gets killed.
In other words, it is leaking.

So, paraphrasing someone else : don't treat the symptom, treat the cause.

>
> However, the memory allocated will get freed up or re-used by the next
> request -

If it is a real leak, then no, it will not.

I think the memory is just fragmented enough to be
> automatically reclaimed by the memory allocator (I've heard that some
> mallocs can return memory to the OS in 1 MB chunks.)

See William's answer : unlikely.

>
> Are there any special techniques people use to avoid this situation?
> Does SizeLimit count actual memory used or does it just look at the
> process size?
>

In a previous similar exercise, in despair I used the module Devel::Leak
as follows :

use Devel::Leak;
my $DEBUGMem = 1;
my ($SVTable,$prevSVCount,$lastSVCount);

if ($DEBUGMem) {
$prevSVCount = Devel::Leak::NoteSV($SVTable);
warn "[$$] before something, total SVs : $prevSVCount";
}

do_something(); # .. which could be leaking

if ($DEBUGMem) {
$lastSVCount = Devel::Leak::CheckSV($SVTable);
warn "[$$] after something : total SVs : $lastSVCount";
warn "[$$] new SVs : ",($lastSVCount - $prevSVCount);
}

It doesn't require any specially-compiled perl.
It does not actually print the memory size used. It just provides a
count of new "things" that have been allocated and not freed by
do_something(). It is very rough, but it was very helpful to me to find
out what exact piece of code was leaking "things", which is basically an
alias for memory.
The point is, if it keeps on growing around the same piece of code each
time you process a request, then you at least know where bad things happen.

If it happens in your code, then when you know where, it should be
possible to fix it. If it happens in someone else's module that you are
using, there are usually several alternative modules for just about
anything on CPAN. If there aren't and you cannot do without, /then/
maybe you should considering limiting the number of requests that each
child handles before it gets killed. But that should not be the first
choice, because it is the least efficient.

Re: avoiding child death by size limit

On Fri, Dec 11, 2009 at 11:37 AM, William T wrote:
> You can make sure the variables that the memory was malloc'd for have
> gone out of scope, and there are not trailing references to them.
> Perl will then reuse that memory.

It will keep the memory allocated to the out-of-scope variables unless
you undef them.

There's a summary of many PerlMonks discussions on this topic here:
http://www.perlmonks.org/?node_id=803515

- Perrin

Re: avoiding child death by size limit

Perrin Harkins wrote:
> On Fri, Dec 11, 2009 at 11:37 AM, William T wrote:
>> You can make sure the variables that the memory was malloc'd for have
>> gone out of scope, and there are not trailing references to them.
>> Perl will then reuse that memory.
>
> It will keep the memory allocated to the out-of-scope variables unless
> you undef them.
>
> There's a summary of many PerlMonks discussions on this topic here:
> http://www.perlmonks.org/?node_id=803515
>
Perrin,
that is in the end, in my personal opinion, a rather confusing discussion.

So, to an extent, is your phrase above.
When you say "It (perl) will keep the memory", do you mean that
- the perl interpreter embedded in this Apache child will keep the
memory (and not return it to the OS), but will re-use it if possible for
other variable allocations happening within its lifetime ?
or
- the perl interpreter embedded in this Apache child will keep the
memory (and not return it to the OS), and never re-use it again for any
other variable allocation happening within its lifetime
(in other words, this is a leak) ?

Does any guru care to provide some simple real-world examples of when
memory once allocated to a variable is/is not being re-used, in a
mod_perl handler context ?
Or pointers to ditto ?
(Maybe at first independently of whether the memory also may, or not, be
returned by Perl to the OS)

Maybe on this last subject, what I gather from this and other
discussions I've seen, is that once the Perl interpreter obtained some
memory from the OS, it rarely returns it (before it exits); and if it
does, it is in any case not practically predictable in a cross-platform
way, so one cannot rely on it. Is that a fair interpretation ?


Would it be preferable/easier if I construct some simple examples myself
and present them here asking if memory allocated to "my $a" is being
leaked or not ?

Re: avoiding child death by size limit

On Fri, Dec 11, 2009 at 5:56 PM, Andr=E9 Warnier wrote:
> When you say "It (perl) will keep the memory", do you mean that
> - the perl interpreter embedded in this Apache child will keep the memory
> (and not return it to the OS), but will re-use it if possible for other
> variable allocations happening within its lifetime ?
> or
> - the perl interpreter embedded in this Apache child will keep the memory
> (and not return it to the OS), and never re-use it again for any other
> variable allocation happening within its lifetime
> (in other words, this is a leak) ?

Option 3:
Perl will keep the memory and reuse it for that exact same lexical
variable the next time you enter that section of code. It's a
performance optimization that is usually a good thing, unless you put
a lot of data in one lexical in some code that you rarely run.

It's not a leak. Perl tracks the memory and will reuse it, just not
for any other variables.

> Does any guru care to provide some simple real-world examples of when mem=
ory
> once allocated to a variable is/is not being re-used, in a mod_perl handl=
er
> context ?

It's simple to see. Slurp a file into a lexical. Let it go out of
scope. There are many posts on this subject in the archives here as
well as on PerlMonks and the p5p archives.

> Maybe on this last subject, what I gather from this and other discussions
> I've seen, is that once the Perl interpreter obtained some memory from th=
e
> OS, it rarely returns it (before it exits); and if it does, it is in any
> case not practically predictable in a cross-platform way, so one cannot r=
ely
> on it. =A0Is that a fair interpretation ?

Yes, you can't expect to get memory back.

- Perrin

Re: avoiding child death by size limit

On Saturday 12 December 2009 00:24:24 Perrin Harkins wrote:
> Perl will keep the memory and reuse it for that exact same lexical
> variable the next time you enter that section of code. It's a
> performance optimization that is usually a good thing, unless you put
> a lot of data in one lexical in some code that you rarely run.
>
perhaps an example can help to shed some light. To understand it you have to
know that many malloc implementations allocate large chunks of memory via
mmap() while smaller pieces are allocated via brk(). The mmap-allocated
blocks can later be returned to the OS while the brk-allocated can not.

So, here is the example. It creates a large string (10 mbytes) and assigns it
to the variable $x. This means another 10mb chunk is allocated to keep the
variable. Devel::Peek::Dump is used to show the address of the string assigned
to the variable.

Now look what is when freed:

strace perl -MDevel::Peek -e '{my $x="x"x(10*1024*1024); $x=""; Dump $x; undef $x; warn 1;} warn 2'

Here 2 memory blocks are allocated. One is the "x"x(10*1024*1024) string the
other is the PV-member of $x. It seems to be logical that the second one is
$x but let's see ...

mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff87f08a000
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff87e689000

These write()s come from Dump $x. The PV=0x7ff87e689028 line below is of
particular interest. It shows that the memory block at 0x7ff87e689000
in fact belongs to $x.

write(2, "SV = ", 5SV = ) = 5
write(2, "PV(0x78bcb0) at 0x78ef78\n REFCN"..., 65PV(0x78bcb0) at 0x78ef78
REFCNT = 1
FLAGS = (PADMY,POK,pPOK)
) = 65
write(2, " PV = 0x7ff87e689028 ", 22 PV = 0x7ff87e689028 ) = 22
write(2, "\"\"\\0", 4""\0) = 4
write(2, "\n", 1
) = 1
write(2, " CUR = 0\n", 10 CUR = 0
) = 10
write(2, " LEN = 10485768\n", 17 LEN = 10485768
) = 17

The next syscall comes from undef $x. Here the second allocated block is
freed and returned to the OS.

munmap(0x7ff87e689000, 10489856) = 0

This write() is the "warn 1"

write(2, "1 at -e line 1.\n", 161 at -e line 1.
) = 16

And this the "warn 2". You see the memory block allocated to keep the
"x"x(10*1024*1024) string is still kept by the program even though the
scope where it has been allocated is finished.

write(2, "2 at -e line 1.\n", 162 at -e line 1.
) = 16

Only right before finishing the program the "x"x(10*1024*1024) string
is freed.

munmap(0x7ff87f08a000, 10489856) = 0
exit_group(0) = ?


Now, omit the "undef $x" and see when the memory allocated to the variable
is freed.

Torsten