Debian Lenny + MaxRequestsPerChild

--001636284d1c19f4c504918da4a4
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

We just upgraded to Debian Lenny and saw some unexpected behaviour from an
Apache node running a mod_perl2 app which I hope someone here can explain.

After running the system for a short while, the server stops accepting
requests. Checking the system, we can see that all workers have disappeared
but the apache2 parent process remains alive. There's no relevant
information in the access or error logs. After some experimentation, we have
also found that if we restart the server with MaxRequestsPerChild set to
zero, it seems to keep going. It thus looks like the workers stop after
serving MaxRequestsPerChild, then are not restarted.

So, any ideas about what's going on or how to troubleshoot this would be
appreciated.

Here are some further details:

apache2.conf - Problematic config section (migrated from etch):

StartServers 2
MaxClients 200
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 50
MaxRequestsPerChild 10000


apache2.conf - working (default) config:

StartServers 2
MaxClients 200
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0


$ apache2 -v
Server version: Apache/2.2.9 (Debian)
Server built: Apr 20 2010 15:42:00

$ APACHE_RUN_USER=www-data APACHE_RUN_GROUP=www-data apache2 -t -D
DUMP_MODULES
Loaded Modules:
core_module (static)
log_config_module (static)
logio_module (static)
mpm_worker_module (static)
http_module (static)
so_module (static)
alias_module (shared)
apreq_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_file_module (shared)
authz_default_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgid_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
deflate_module (shared)
env_module (shared)
headers_module (shared)
mime_module (shared)
negotiation_module (shared)
perl_module (shared)
proxy_module (shared)
proxy_http_module (shared)
setenvif_module (shared)
ssl_module (shared)
status_module (shared)
Syntax OK

Best regards,
Thomas
--
Thomas Lindgren, Chief Technology Officer, Diino AB

--001636284d1c19f4c504918da4a4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi all,

We just upgraded to Debian Lenny and saw some unexpected beh=
aviour from an Apache node running a mod_perl2 app which I hope someone her=
e can explain.

After running the system for a short while, the serv=
er stops accepting requests. Checking the system, we can see that all worke=
rs have disappeared but the apache2 parent process remains alive. There'=
;s no relevant information in the access or error logs. After some experime=
ntation, we have also found that if we restart the server with MaxRequestsP=
erChild set to zero, it seems to keep going. It thus looks like the workers=
stop after serving MaxRequestsPerChild, then are not restarted.



So, any ideas about what's going on or how to troubleshoot this wou=
ld be appreciated.

Here are some further details:

apache2.con=
f - Problematic config section (migrated from etch):


<IfModule mpm_worker_module>


=A0 =A0StartServers =A0 =A0 =A0 =A0 =A02


=A0 =A0MaxClients =A0 =A0 =A0 =A0 =A0200


=A0 =A0MinSpareThreads =A0 =A0 =A025


=A0 =A0MaxSpareThreads =A0 =A0 =A075


=A0 ThreadsPerChild =A0 =A0 =A050


=A0 MaxRequestsPerChild =A0 10000


</IfModule>





apache2.conf - working (default) config:


<IfModule mpm_worker_module>


=A0 =A0StartServers =A0 =A0 =A0 =A0 =A02


=A0 =A0MaxClients =A0 =A0 =A0 =A0 =A0200


=A0 =A0MinSpareThreads =A0 =A0 =A025


=A0 =A0MaxSpareThreads =A0 =A0 =A075


=A0 ThreadsPerChild =A0 =A0 =A025


=A0 MaxRequestsPerChild =A0 0


</IfModule>



$ apache2 -v

Server version: Apache/2.2.9 (Debian)

Server built: =A0 Apr 20 2010 15:42:00



$ APACHE_RUN_USER=3Dwww-data APACHE_RUN_GROUP=3Dwww-data apache2 -t -D=20
DUMP_MODULES

Loaded Modules:

=A0core_module (static)

=A0log_config_module (static)

=A0logio_module (static)

=A0mpm_worker_module (static)

=A0http_module (static)

=A0so_module (static)

=A0alias_module (shared)

=A0apreq_module (shared)

=A0auth_basic_module (shared)

=A0auth_digest_module (shared)

=A0authn_file_module (shared)

=A0authz_default_module (shared)

=A0authz_groupfile_module (shared)

=A0authz_host_module (shared)

=A0authz_user_module (shared)

=A0autoindex_module (shared)

=A0cgid_module (shared)

=A0dav_module (shared)

=A0dav_fs_module (shared)

=A0dav_lock_module (shared)

=A0deflate_module (shared)

=A0env_module (shared)

=A0headers_module (shared)

=A0mime_module (shared)

=A0negotiation_module (shared)

=A0perl_module (shared)

=A0proxy_module (shared)

=A0proxy_http_module (shared)

=A0setenvif_module (shared)

=A0ssl_module (shared)

=A0status_module (shared)

Syntax OK



Best regards,
Thomas
--
Thomas Lindgren, Chief Technology Officer=
, Diino AB



--001636284d1c19f4c504918da4a4--

Re: Debian Lenny + MaxRequestsPerChild

----- "Thomas Lindgren" wrote:

> Hi all,
>=20
> We just upgraded to Debian Lenny and saw some unexpected behaviour
> from an Apache node running a mod_perl2 app which I hope someone here
> can explain.
>=20
> After running the system for a short while, the server stops accepting
> requests. Checking the system, we can see that all workers have
> disappeared but the apache2 parent process remains alive. There's no
> relevant information in the access or error logs. After some
> experimentation, we have also found that if we restart the server with
> MaxRequestsPerChild set to zero, it seems to keep going. It thus looks
> like the workers stop after serving MaxRequestsPerChild, then are not
> restarted.
>=20
> So, any ideas about what's going on or how to troubleshoot this would
> be appreciated.

Can you get a strace from the parent once, or shortly before it reaches
such a state?

gdb would also be a plus...

gcore $pid etc..

It could also be that some of Debian's patches are causing this..

you could try to compile (the latest versions of) httpd and mod_perl2
and see if you can reproduce this behaviour.

(See https://scm.brainsware.org/svn/webstack/linux/Makefile on how
to compile for Debian)

But before going down that rourte ith might be worth elaborating
what your httpd does, other then serve a mod_perl2 application.
Since you do have a number of other modules loaded which could
be useful or suspects in this case.

> Here are some further details:
>=20
> apache2.conf - Problematic config section (migrated from etch):
>
> StartServers 2
> MaxClients 200
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadsPerChild 50
> MaxRequestsPerChild 10000
>
>=20
> apache2.conf - working (default) config:
>
> StartServers 2
> MaxClients 200
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadsPerChild 25
> MaxRequestsPerChild 0
>
>=20
> $ apache2 -v
> Server version: Apache/2.2.9 (Debian)
> Server built: Apr 20 2010 15:42:00
>=20
> $ APACHE_RUN_USER=3Dwww-data APACHE_RUN_GROUP=3Dwww-data apache2 -t -D
> DUMP_MODULES
> Loaded Modules:
> core_module (static)
> log_config_module (static)
> logio_module (static)
> mpm_worker_module (static)
> http_module (static)
> so_module (static)
> alias_module (shared)
> apreq_module (shared)
> auth_basic_module (shared)
> auth_digest_module (shared)
> authn_file_module (shared)
> authz_default_module (shared)
> authz_groupfile_module (shared)
> authz_host_module (shared)
> authz_user_module (shared)
> autoindex_module (shared)
> cgid_module (shared)
> dav_module (shared)
> dav_fs_module (shared)
> dav_lock_module (shared)
> deflate_module (shared)
> env_module (shared)
> headers_module (shared)
> mime_module (shared)
> negotiation_module (shared)
> perl_module (shared)
> proxy_module (shared)
> proxy_http_module (shared)
> setenvif_module (shared)
> ssl_module (shared)
> status_module (shared)
> Syntax OK
>=20
> Best regards,
> Thomas
> --
> Thomas Lindgren, Chief Technology Officer, Diino AB

--=20
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Debian Lenny + MaxRequestsPerChild

--0016364ee260a1b5f70491cc00b7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

2010/10/2 Igor Galić

>
> ----- "Thomas Lindgren" wrote:
>
> > Hi all,
> >
> > We just upgraded to Debian Lenny and saw some unexpected behaviour
> > from an Apache node running a mod_perl2 app which I hope someone here
> > can explain.
> >
> > After running the system for a short while, the server stops accepting
> > requests. Checking the system, we can see that all workers have
> > disappeared but the apache2 parent process remains alive. There's no
> > relevant information in the access or error logs. After some
> > experimentation, we have also found that if we restart the server with
> > MaxRequestsPerChild set to zero, it seems to keep going. It thus looks
> > like the workers stop after serving MaxRequestsPerChild, then are not
> > restarted.
> >
> > So, any ideas about what's going on or how to troubleshoot this would
> > be appreciated.
>
> Can you get a strace from the parent once, or shortly before it reaches
> such a state?
>
> gdb would also be a plus...
>
> gcore $pid etc..
>
> It could also be that some of Debian's patches are causing this..
>
> you could try to compile (the latest versions of) httpd and mod_perl2
> and see if you can reproduce this behaviour.
>
> (See https://scm.brainsware.org/svn/webstack/linux/Makefile on how
> to compile for Debian)
>
> But before going down that rourte ith might be worth elaborating
> what your httpd does, other then serve a mod_perl2 application.
> Since you do have a number of other modules loaded which could
> be useful or suspects in this case.
>
>
Hi Igor,

Thanks for the tips, we'll try to reproduce this first with strace/gdb
active. Second, checking this with a clean compile might be a good idea too=
..
The mod_perl2 app is mainly used for extended/hacked WebDAV-access and has
been in production for a couple of years under Debian Etch (where we haven'=
t
seen this sort of behaviour before).

By the way, the version with MaxRequestsPerChild =3D 0 stayed up over the
weekend, so it seems to be a workaround of sorts. But we'd prefer to use
the original config, so we'll do some more investigating even so.

Best regards,
Thomas
--=20
Thomas Lindgren, Chief Technology Officer, Diino AB

--0016364ee260a1b5f70491cc00b7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Re: Debian Lenny + MaxRequestsPerChild

> Hi Igor,
>=20
> Thanks for the tips, we'll try to reproduce this first with strace/gdb
> active. Second, checking this with a clean compile might be a good
> idea too. The mod_perl2 app is mainly used for extended/hacked
> WebDAV-access and has been in production for a couple of years under
> Debian Etch (where we haven't seen this sort of behaviour before).

So much for Debian ``stable''...
=20
> By the way, the version with MaxRequestsPerChild =3D 0 stayed up over
> the weekend, so it seems to be a workaround of sorts. But we'd prefer
> to use the original config, so we'll do some more investigating even
> so.

Generally: You don't need to recycle children, unless they leak memory.
That is exactly what MaxRequestsPerChild (should be MaxConnectionsPerChild
actually) does.

> Best regards,
> Thomas
> --
> Thomas Lindgren, Chief Technology Officer, Diino AB

Bye,
i
--=20
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org