apache over ServerLimit processes limit

hi all,

we have busy web site and for last few weeks we have problem with apache
processes eat all memory (that is problem, but not topic here).
I attempt to control problem by lowering ServerLimit (documentation states
that is max number of apache processes).

ServerLimit is at 75 (confirmed by /server-info as active value), but 'ps auxw'
show around 90-95 processes. process creation times are within 10 minutes.
i was try to report that as bug (in my view it is), but i got redirected here. :)

here is part of server config as-is:


ServerLimit 75
ThreadLimit 30
StartServers 11
MinSpareThreads 45
MaxSpareThreads 95
ThreadsPerChild 25
MaxClients 1500
MaxRequestsPerChild 500




tech info:
server is 12core, 8Gb ram. cpu usage is about 20-25% under normal load. memory is
about 45-50% in use at busier part of day.
apache is 2.2.14, gcc 4.3.4, gentoo distribution. php is 5.2.12.


problem development: at some point in time processes start to eat more memory (all, not single one),
and server locks after 3-5 minutes (no swap configured as this make stuff even more painful).
if i grab some server statistics (ps, netstat, df...) between start of problem and lock-up, i
see there are more apache processes than configured to be.

is this a bug in ServerLimit or that directive really cannot keep server
processes limited to number?


fli


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: apache over ServerLimit processes limit

On Mon, Feb 15, 2010 at 7:19 AM, Zdravko Spoljar
wrote:
>
> hi all,
>
> we have busy web site and for last few weeks we have problem with apache
> processes eat all memory (that is problem, but not topic here).
> I attempt to control problem by lowering ServerLimit (documentation state=
s
> that is max number of apache processes).
>
> ServerLimit is at 75 (confirmed by /server-info as active value), but 'ps=
auxw'
> show around 90-95 processes. process creation times are within 10 minutes=
..
> i was try to report that as bug (in my view it is), but i got redirected =
here. :)

Read the fourth paragraph at

http://httpd.apache.org/docs/2.2/mod/worker.html#how-it-work s

> here is part of server config as-is:
>
>
> ServerLimit 75
> ThreadLimit 30
> =A0 =A0 =A0 =A0StartServers =A0 =A0 =A0 =A0 =A0 =A011
> =A0 =A0 =A0 =A0MinSpareThreads =A0 =A0 =A0 =A0 45
> =A0 =A0 =A0 =A0MaxSpareThreads =A0 =A0 =A0 =A0 95
> =A0 =A0 =A0 =A0ThreadsPerChild =A0 =A0 =A0 =A0 25
> =A0 =A0 =A0 =A0MaxClients =A0 =A0 =A0 =A0 =A0 =A0 =A01500
> =A0 =A0 =A0 =A0MaxRequestsPerChild =A0 =A0 500
>

Anything that causes child processes to exit during steady state can
lead to extra child processes *when some requests take a relatively
long time to finish*.

Your very low value for MaxRequestsPerChild is the likely culprit.
You might have to increase MaxSpareThreads as well since that will
cause one or more child processes to begin termination when load
decreases by a moderate amount.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: apache over ServerLimit processes limit

On 02/15/2010 05:49 PM, Zdravko Spoljar wrote:
>
> hi all,
>
> we have busy web site and for last few weeks we have problem with apache
> processes eat all memory (that is problem, but not topic here).
> I attempt to control problem by lowering ServerLimit (documentation states
> that is max number of apache processes).
>
> ServerLimit is at 75 (confirmed by /server-info as active value), but 'ps auxw'
> show around 90-95 processes. process creation times are within 10 minutes.
> i was try to report that as bug (in my view it is), but i got redirected here. :)
>
> here is part of server config as-is:
>
>
> ServerLimit 75
> ThreadLimit 30
> StartServers 11
> MinSpareThreads 45
> MaxSpareThreads 95
> ThreadsPerChild 25
> MaxClients 1500
> MaxRequestsPerChild 500
>
>
>
>
> tech info:
> server is 12core, 8Gb ram. cpu usage is about 20-25% under normal load. memory is
> about 45-50% in use at busier part of day.
> apache is 2.2.14, gcc 4.3.4, gentoo distribution. php is 5.2.12.
>
>
> problem development: at some point in time processes start to eat more memory (all, not single one),
> and server locks after 3-5 minutes (no swap configured as this make stuff even more painful).
> if i grab some server statistics (ps, netstat, df...) between start of problem and lock-up, i
> see there are more apache processes than configured to be.
>
> is this a bug in ServerLimit or that directive really cannot keep server
> processes limited to number?
>
>
> fli
>
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>

Are you sure that you're using worker MPM ?

--
Nilesh Govindarajan
Site & Server Adminstrator
www.itech7.com

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: apache over ServerLimit processes limit

On Mon, 2010-02-15 at 07:29 -0500, Jeff Trawick wrote:



>
> Anything that causes child processes to exit during steady state can
> lead to extra child processes *when some requests take a relatively
> long time to finish*.

now, if i understand correctly "active child processes" != "child
processes which are terminating" so relative long lived request hold
additional childs alive, and apache spawns new one above ServerLimits to
server new requests. And there is no directive which will limit number
of apache processes in absolute sense? so only solution would be to use
kernel mechanism for limiting number of processes for apache user.
of course that can lead to "worst case" situation in which i will have
up to 1 per child slow request finishing, and no "fast" traffic served
because there are no available threads.
right? :-)

> Your very low value for MaxRequestsPerChild is the likely culprit.
> You might have to increase MaxSpareThreads as well since that will
> cause one or more child processes to begin termination when load
> decreases by a moderate amount.

that was in because i wanted relative short time of recycling processes,
to avoid memleaks, but i get short of memory due long live requests :-(
(that is relatively new problem, as php application works for over year)

to summ:
i should rise MaxRequestsPerChild to 5000 (or more), and have slower
rate of recycling processes? and setup some way to kill off long live
requests (maybe killed by firewall setup or by some mod_qos options).
also setup max number of processes in kernel (and max process RSS size)
to stabilize server (as apache cannot take care of itself).

i would rather have locked/crashed apache (which can be detected&fixed
by cron job) than have unusable system due lack of memory (which i need
to fix myself :)

any way, thanks for precise point to my lack of understanding of
implications of using MaxRequestsPerChild != 0.



fli



------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: apache over ServerLimit processes limit

On Mon, 2010-02-15 at 19:00 +0530, Nilesh Govindarajan wrote:

>
> Are you sure that you're using worker MPM ?
>

i have (in Gentoo):
APACHE2_MPMS = "worker" and USE flag "threads" so yes, i'm
100% sure this is worker MPM in use.
also 'ps auxm' show multiple threads per process, and number of
processes is far less than number of active connections to clients. :)



fli


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: apache over ServerLimit processes limit

On Mon, Feb 15, 2010 at 9:01 AM, Zdravko Spoljar
wrote:
> On Mon, 2010-02-15 at 07:29 -0500, Jeff Trawick wrote:
>
>
>
>>
>> Anything that causes child processes to exit during steady state can
>> lead to extra child processes *when some requests take a relatively
>> long time to finish*.
>
> now, if i understand correctly "active child processes" != "child
> processes which are terminating" so relative long lived request hold
> additional childs alive, and apache spawns new one above ServerLimits to
> server new requests. And there is no directive which will limit number
> of apache processes in absolute sense?

Absolute control

Use this combination of directives:

MaxSpareThreads =MaxClients
MaxRequestsPerChild 0

Since graceful restart has the same effect -- telling a child process
to terminate even though the requests it is handling won't finish for
a long time -- you can't use graceful restart either.

Not absolute control but still work-able with a relatively small
number of extra child processes as long as individual requests don't
hang:

MaxSpareThreads not small enough that parent terminates child
processes except in rare circumstances (e.g., after a period of
abnormally high client connections)

MaxRequestsPerChild enabled, but only after a large number of requests
(e.g., 80000)

When you don't use absolute control, you'll need to monitor behavior
for a while to see if MaxSpareThreads or MaxRequestsPerChild needs to
be increased.

> so only solution would be to use
> kernel mechanism for limiting number of processes for apache user.
> of course that can lead to "worst case" situation in which i will have
> up to 1 per child slow request finishing, and no "fast" traffic served
> because there are no available threads.
> right? :-)

(Solution above)

That's why worker allows these extra processes to be created.

>
>> Your very low value for MaxRequestsPerChild is the likely culprit.
>> You might have to increase MaxSpareThreads as well since that will
>> cause one or more child processes to begin termination when load
>> decreases by a moderate amount.
>
> that was in because i wanted relative short time of recycling processes,
> to avoid memleaks, but i get short of memory due long live requests :-(
> (that is relatively new problem, as php application works for over year)
>
> to summ:
> i should rise MaxRequestsPerChild to 5000 (or more),

I'd guess higher, but check watching the number of extra processes

> and have slower
> rate of recycling processes?

yes

> and setup some way to kill off long live
> requests (maybe killed by firewall setup or by some mod_qos options).

investigate why they are long running; if because of client behavior
(very slow upload of large data), then firewall setup could help; but
often it is because some application logic is hung, and Apache will
wait forever for mod_php to return control

> also setup max number of processes in kernel

this should only be plan B just in case tuning of MaxSpareThreads and
MaxRequestsPerChild doesn't handle some rare condition

> (and max process RSS size)

one thread going over the limit will trash all other requests being
served by that process, right?

> to stabilize server (as apache cannot take care of itself).

because of the combination of threaded MPM, the need for aggressive
child process termination because of (presumably) an application
problem, and long-running requests

>
> i would rather have locked/crashed apache (which can be detected&fixed
> by cron job) than have unusable system due lack of memory (which i need
> to fix myself :)
>
> any way, thanks for precise point to my lack of understanding of
> implications of using MaxRequestsPerChild != 0.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org