huge apache+mod_perl processes

This is a multi-part message in MIME format.

------=_NextPart_000_0102_01CB602C.7997A540
Content-Type: text/plain;
charset="koi8-r"
Content-Transfer-Encoding: quoted-printable

Greetings,

We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb in =
size. After apache restart they are usually 40-50 Mb in size, then in a =
minute grow up to 100-150 Mb and then some time later may grow up to 200 =
Mb. I suspect a certain type of http queries and would like to know if =
there are any manuals/howtos/tools to investiagate such cases.

# perl -v
This is perl, v5.10.1 (*) built for x86_64-linux

# strings /usr/local/apache2/modules/mod_perl.so | grep mod_perl\/
mod_perl/2.0.4

# /usr/local/apache2/bin/httpd -v
Server version: Apache/2.2.14 (Unix)

# pmap -x 8320
8320: /usr/local/apache2/bin/httpd -DSSL
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 584 496 0 r-x-- httpd
0000000000692000 24 20 20 rw--- httpd
0000000000698000 12 8 8 rw--- [ anon ]
0000000017c10000 147640 144612 144608 rw--- [ anon ]
.....
---------------- ------ ------ ------
total kB 352380 156560 151324


# /usr/local/apache2/bin/httpd -l
Compiled in modules:
core.c
mod_authn_file.c
mod_authn_default.c
mod_authz_host.c
mod_authz_groupfile.c
mod_authz_user.c
mod_authz_default.c
mod_auth_basic.c
mod_include.c
mod_filter.c
mod_log_config.c
mod_env.c
mod_setenvif.c
mod_version.c
prefork.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgi.c
mod_negotiation.c
mod_dir.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_rewrite.c
mod_so.c
#


Cheers
Eugene
------=_NextPart_000_0102_01CB602C.7997A540
Content-Type: text/html;
charset="koi8-r"
Content-Transfer-Encoding: quoted-printable

Re: huge apache+mod_perl processes

--Apple-Mail-1-434063194
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii


On 29 Sep 2010, at 20:17, Eugene Toropov wrote:

> Greetings,
> =20
> We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb =
in size. After apache restart they are usually 40-50 Mb in size, then in =
a minute grow up to 100-150 Mb and then some time later may grow up to =
200 Mb. I suspect a certain type of http queries and would like to know =
if there are any manuals/howtos/tools to investiagate such cases.
> =20

Yes.

http://perl.apache.org/docs/1.0/guide/performance.html

Still good for 2.0.



--Apple-Mail-1-434063194
Content-Disposition: attachment;
filename=smime.p7s
Content-Type: application/pkcs7-signature;
name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEH AQAAoIIDWjCCA1Yw
ggI+oAMCAQICAQEwCwYJKoZIhvcNAQEFMEoxGDAWBgNVBAMMD0RhdmUgSG9k Z2tpbnNvbjELMAkG
A1UEBhMCR0IxITAfBgkqhkiG9w0BCQEWEmRhdmVob2RnQGdtYWlsLmNvbTAe Fw0xMDA1MjYxMjM3
NDVaFw0xMTA1MjYxMjM3NDVaMEoxGDAWBgNVBAMMD0RhdmUgSG9kZ2tpbnNv bjELMAkGA1UEBhMC
R0IxITAfBgkqhkiG9w0BCQEWEmRhdmVob2RnQGdtYWlsLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAKNEsd4Pz8mjFrM97NLC6WwfXMJEFGgw9+1j5RPhVvFC D+jPATGVzON1lVaB
9C5vMQov/hzC6/B0bynpNzQnZC+v3Vy+Flgro+XNh1rMPtqqK757bjwEQk/3 deB3yHuT6qCzj3Mb
ze5uuGYGJFyzOeKjteUp0UqrgiMl587qE1OGSrvMWCKSAo1nNgXA8FXnn/nN jgLrocwM8GrzMfvG
pA3bpjdKeZxSN4KgWnHZNYMb50CNOs6epET0snvEpzjojjhCdHTQUWZ+FZqF nN8aNmm2/hB/D2vO
8XBF+wNzsbljDXxL7wvgJEIqvvEFhvRYzJfkC5iWsSfss5nxeNINV2MCAwEA AaNJMEcwDgYDVR0P
AQH/BAQDAgeAMBYGA1UdJQEB/wQMMAoGCCsGAQUFBwMEMB0GA1UdEQQWMBSB EmRhdmVob2RnQGdt
YWlsLmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAiwQ36Blz4Ud5zViOOt13qTf6 trx+mD2Q4a21C7rJ
WOjSclwaCa97VW9ZNxdXkPTG+QQY0kDNNEp601rNbmRUoSoHOAKtWwdeDL7C U1jStW5Wld6443Bk
lgepnkuK5Wgau0FZo20L9seKWkG1A9PpbanwSsyazVaPl8hNVzao5SCjvly0 9x9rE2ba4uv05bZj
0AZbOjKX25cNoBlc8YBgrdY0UpMz8tMwQEeIlJtzzF/YN3Hcvd279wSuVRZK j0UCufduHILVkdit
xWKgWMKRoGbBuV6KUQEs8JR6XwHq6bSCu326MIMdSx7HpqCfr+BtLeeCy6Cm 58o/jyt9cpFgUTGC
ApkwggKVAgEBME8wSjEYMBYGA1UEAwwPRGF2ZSBIb2Rna2luc29uMQswCQYD VQQGEwJHQjEhMB8G
CSqGSIb3DQEJARYSZGF2ZWhvZGdAZ21haWwuY29tAgEBMAkGBSsOAwIaBQCg ggEfMBgGCSqGSIb3
DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDkzMDA3MjM0 NVowIwYJKoZIhvcN
AQkEMRYEFCKgG7DL2CYpZbigSlo43+b0ETTsMF4GCSsGAQQBgjcQBDFRME8w SjEYMBYGA1UEAwwP
RGF2ZSBIb2Rna2luc29uMQswCQYDVQQGEwJHQjEhMB8GCSqGSIb3DQEJARYS ZGF2ZWhvZGdAZ21h
aWwuY29tAgEBMGAGCyqGSIb3DQEJEAILMVGgTzBKMRgwFgYDVQQDDA9EYXZl IEhvZGdraW5zb24x
CzAJBgNVBAYTAkdCMSEwHwYJKoZIhvcNAQkBFhJkYXZlaG9kZ0BnbWFpbC5j b20CAQEwDQYJKoZI
hvcNAQEBBQAEggEAQcVmTG8CJl+3RIeHSeWKt3bNYti3TOlPqcq1bfSK+Y4n loH2q2ykRO+8cN7i
LYEhTY9TarnPCIA3YDNbaTSuy7UOwAII6W9DmAxcRAGlIc0al8TE0zwq5+aF K48LiEj72oujbT9m
yM896ju692rFez7EHXVvfSMNiXaIs3e7INI4wVgWglV1tsrYC3zIRPeHdrCq 0k59/4Q9B7XDENFd
kfxm3OIe42yWEhzUyVJq5Mfzn+4k74ioArrC3xCjUyAm4veoHnCVI4CjCeWY EfyAhh+GpzWrZH16
cGst4oJUgkg7Zy2AWa52JWuCrlG2HVRRfxRwHTsEVDZ1U+7Id/2d4QAAAAAA AA==

--Apple-Mail-1-434063194--

Re: huge apache+mod_perl processes

Eugene Toropov wrote:
> Greetings,
>
> We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb in size. After apache restart they are usually 40-50 Mb in size, then in a minute grow up to 100-150 Mb and then some time later may grow up to 200 Mb.

One basic characteristic of Perl's memory management is that, once it has acquired a
certain amount of memory from the OS under which it runs, it never gives it back to the OS.
For stand-alone perl scripts, this rarely matters, as when the script terminates, the perl
interpreter process also terminates, and the memory it used is then returned to the OS.
But under apache/mod_perl, the same perl interpreter stays alive as long as the apache
process (e.g., an apache child process) in which it runs is alive.
So over time, if your perl scripts/modules are not a bit careful, the amount of memory
used can grow and grow.
Inside of perl itself, the memory used by scripts and modules is normally reclaimed and
recycled. But perl's "garbage collector" can only recycle the memory used by a variable
if there are no more references to that variable.

Re: huge apache+mod_perl processes

If you don't have an unbounded growth issue it is likely do to some
library pulling in alot of dependencies or the creation/caching in
memory of some large data structure. You can preload all the
offending libraries and see if that causes a jump in the initial
memory allocation for you apache procs. If not you can create a
Handler that fires during cleanup/log phase to capture memory size
right after each access.

2010/9/29 Eugene Toropov :
> Greetings,
>
> We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb in
> size. After apache restart they are usually 40-50 Mb in size, then in a
> minute grow up to 100-150 Mb=A0and then some time later may grow up to 20=
0 Mb.
> I suspect a certain type of http queries=A0and would like to know if ther=
e are
> any manuals/howtos/tools to investiagate such cases.
>
> # perl -v
> This is perl, v5.10.1 (*) built for x86_64-linux
> # strings /usr/local/apache2/modules/mod_perl.so | grep mod_perl\/
> mod_perl/2.0.4
> # /usr/local/apache2/bin/httpd -v
> Server version: Apache/2.2.14 (Unix)
>
> # pmap -x 8320
> 8320:   /usr/local/apache2/bin/httpd -DSSL
> Address           Kbytes     RSS   Dirty =
Mode   Mapping
> 0000000000400000     584     496       0 r-x-=
-=A0 httpd
> 0000000000692000    =A0 24    =A0 20    =A0 20 rw=
---=A0 httpd
> 0000000000698000    =A0 12       8       =
8 rw---  =A0 [ anon ]
> 0000000017c10000=A0 147640=A0 144612=A0 144608 rw---  =A0 [ anon ]
> ....
> ----------------=A0 ------=A0 ------=A0 ------
> total kB        =A0 352380=A0 156560=A0 151324
>
> # /usr/local/apache2/bin/httpd -l
> Compiled in modules:
> =A0 core.c
> =A0 mod_authn_file.c
> =A0 mod_authn_default.c
> =A0 mod_authz_host.c
> =A0 mod_authz_groupfile.c
> =A0 mod_authz_user.c
> =A0 mod_authz_default.c
> =A0 mod_auth_basic.c
> =A0 mod_include.c
> =A0 mod_filter.c
> =A0 mod_log_config.c
> =A0 mod_env.c
> =A0 mod_setenvif.c
> =A0 mod_version.c
> =A0 prefork.c
> =A0 http_core.c
> =A0 mod_mime.c
> =A0 mod_status.c
> =A0 mod_autoindex.c
> =A0 mod_asis.c
> =A0 mod_cgi.c
> =A0 mod_negotiation.c
> =A0 mod_dir.c
> =A0 mod_actions.c
> =A0 mod_userdir.c
> =A0 mod_alias.c
> =A0 mod_rewrite.c
> =A0 mod_so.c
> #
> Cheers
> Eugene

Re: huge apache+mod_perl processes

Also, if you are using something like Cache::FastMmap, your processes
will look quite large depending on the size of your cache, but this
memory is shared between children so it is not that big a deal. From
the docs:

-----
Because Cache::FastMmap mmap's a shared file into your processes
memory space, this can make each process look quite large, even though
it's just mmap'd memory that's shared between all processes that use
the cache, and may even be swapped out if the cache is getting low
usage.
-----

Not sure if you are using this module, but it could explain your
process size jumping up by a large amount, but not growing
significantly after that.

Cheers,

Cees

On Wed, Oct 6, 2010 at 5:49 PM, William T wrote:
> If you don't have an unbounded growth issue it is likely do to some
> library pulling in alot of dependencies or the creation/caching in
> memory of some large data structure. =A0You can preload all the
> offending libraries and see if that causes a jump in the initial
> memory allocation for you apache procs. =A0If not you can create a
> Handler that fires during cleanup/log phase to capture memory size
> right after each access.
>
> 2010/9/29 Eugene Toropov :
>> Greetings,
>>
>> We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb in
>> size. After apache restart they are usually 40-50 Mb in size, then in a
>> minute grow up to 100-150 Mb=A0and then some time later may grow up to 2=
00 Mb.
>> I suspect a certain type of http queries=A0and would like to know if the=
re are
>> any manuals/howtos/tools to investiagate such cases.

Re: huge apache+mod_perl processes

Thank you very much for all replies. We're investigating the issue in our
dev environment with all your notes taken into account. We don't use
Cache::FastMmap. Suspected handlers heavily use Coro/AnyEvent. I'll write
about results once we have finished.

Cheers
Eugene

----- Original Message -----
From: "Cees Hek"
To: "William T"
Cc: "Eugene Toropov" ;
Sent: Thursday, October 07, 2010 3:06 AM
Subject: Re: huge apache+mod_perl processes


Also, if you are using something like Cache::FastMmap, your processes
will look quite large depending on the size of your cache, but this
memory is shared between children so it is not that big a deal. From
the docs:

-----
Because Cache::FastMmap mmap's a shared file into your processes
memory space, this can make each process look quite large, even though
it's just mmap'd memory that's shared between all processes that use
the cache, and may even be swapped out if the cache is getting low
usage.
-----

Not sure if you are using this module, but it could explain your
process size jumping up by a large amount, but not growing
significantly after that.

Cheers,

Cees

On Wed, Oct 6, 2010 at 5:49 PM, William T wrote:
> If you don't have an unbounded growth issue it is likely do to some
> library pulling in alot of dependencies or the creation/caching in
> memory of some large data structure. You can preload all the
> offending libraries and see if that causes a jump in the initial
> memory allocation for you apache procs. If not you can create a
> Handler that fires during cleanup/log phase to capture memory size
> right after each access.
>
> 2010/9/29 Eugene Toropov :
>> Greetings,
>>
>> We have a problem with huge Apache+mod_perl2 processes of 150-200 Mb in
>> size. After apache restart they are usually 40-50 Mb in size, then in a
>> minute grow up to 100-150 Mb and then some time later may grow up to 200
>> Mb.
>> I suspect a certain type of http queries and would like to know if there
>> are
>> any manuals/howtos/tools to investiagate such cases.