Figuring out which perl file is leaking memory

I'm working with some old perl code that was previously running under
modperl 1.x, but was recently updated for 2.x. There is a fairly severe
memory leak somewhere in this code, but I'm having a hell of a time
tracking it down.

The problem is that I don't know which of our many perl files is causing
the leak. I've tried setting up a test server and hitting the likely
suspects with ab, but no dice so far.

So, my question is: is there a way for me to figure out *which perl
file* is causing the leak, without just manually testing every code path
of every file?

-Michael

Re: Figuring out which perl file is leaking memory

William T wrote:
> There are some leak analysis modules that may help (Devel::Leak I
> think). I've tried using them in the past, but they haven't worked
> well for me for various reasons. I found commenting out early branch
> points allowed me (most of the time) to narrow down where the leak was
> actually happening. The I typically look fr anything that looks like
> a closure or some code playing with references.

Rolf Banting wrote:
> There's a few modules on CPAN that might help e.g. Devel::LeakTrace.

The problem is that there are a lot of perl files to check out, and I
don't know which one has the leak. What I'm asking is whether there's a
way to figure out *which file* is leaking memory, without having to test
each file individually. Some kind of per-file memory usage statistics
from modperl or somesuch.

Re: Figuring out which perl file is leaking memory

Michael Gardner wrote:
> William T wrote:
>> There are some leak analysis modules that may help (Devel::Leak I
>> think). I've tried using them in the past, but they haven't worked
>> well for me for various reasons. I found commenting out early branch
>> points allowed me (most of the time) to narrow down where the leak was
>> actually happening. The I typically look fr anything that looks like
>> a closure or some code playing with references.
>
> Rolf Banting wrote:
>> There's a few modules on CPAN that might help e.g. Devel::LeakTrace.
>
> The problem is that there are a lot of perl files to check out, and I
> don't know which one has the leak. What I'm asking is whether there's a
> way to figure out *which file* is leaking memory, without having to test
> each file individually. Some kind of per-file memory usage statistics
> from modperl or somesuch.
>
I don't think there is an /easy/ way, but maybe something along these
lines : I believe mod_perl redefines a number of things, such as exit(),
return() etc.. Maybe you could re-redefine them, and arrange to call
something there which would give you some info ? That could be done,
for example, in your startup.pl script (the one you execute when
starting the server and initially loading mod_perl).

Re: Figuring out which perl file is leaking memory

On Fri, Jun 12, 2009 at 6:00 AM, Michael Gardner wrote:
> The problem is that there are a lot of perl files to check out, and I
> don't know which one has the leak. What I'm asking is whether there's a
> way to figure out *which file* is leaking memory, without having to test
> each file individually. Some kind of per-file memory usage statistics
> from modperl or somesuch.

There are two things you can try. First, use Apache::Status to see
how much memory is being used by different packages. If that doesn't
help, add a log handler to your server that prints the difference in
memory used by the current process at the start of the request and the
end of it. (Capture the size at the start in a fixup handler or
similar.)

- Perrin

Re: Figuring out which perl file is leaking memory

Thanks for the tips. For the record, it turned out that the "leak"
wasn't really a leak at all: we have a *large* directory structure in
which each directory was using as its index file a symlink to the same
perl file (which in turn used the REQUEST_URI to figure out what content
to serve). Apparently mod_perl treats each of those as a separate file,
so it was caching thousands of copies of the same file over time.

The reason I didn't find this sooner is that I had been testing that
index file via only a handful of URIs, and thus didn't see the memory
usage grow. Once I turned to `wget -r', the "leak" finally surfaced. I
fixed it by converting the index file to a real mod_perl handler and
pointing the right URIs at it manually in httpd.conf.

Sorry for the extra noise, but I figured the info might be useful to
somebody. Thanks again, guys.

-Michael