W state, Apache DOS

Hello All,

I would like to share some experience with you as regarded to one cause of an
issue with Apache threads hanging in W state, causing a temporary DOS
of HTTP server.

The issue was tracked down to session_open function and PHP garbage collector.
When you have a relatively busy server with more than 50K hosts hitting PHP pages
and about 200K sessions in tmp folder the behavior leading to W
state is as follows:

* PHP calls session_open
* garbage collector is started
* session_open is locked till GC is finished
* by some reason (not yet investigated) PHP's GC process takes a lot of time, and I mean a lot, a
lot more than 20 minutes, during that time all the subsequent calls of session_open
are locked and waiting till the GC is over
* apache threads are consumed
* apache reaches maxclients
* server resources are consumed but no load is generated
* Apche DOS, no server load is generated it looks like GC just hangs

by some reasons PHP's GC is not affected by execution time
restrictions that means that no matter how long the resource
policies aren't applied to GC and the script will hang in session_open
state almost indefinitely

A simple work around to that problem would be something like:

#find /tmp/* -name "sess_*" -type f -cmin +120 | xargs rm 2>/dev/null

the goal is to keep minimum session files in GC tmp folder

Another workaround would be to turn off PHP's GC completely and do it
on your own.

It is really a bad idea to "hook" system tasks to web calls, like GC
does.

Everyone knows that, why PHP is not applying some program that would
handle GC independently, say as a crontab job, - I can't tell. It's just
wrong by design.

Hope this might shed some light on one cause of Apache's W state.


--
Best regards,
Igor mailto:sprog@online.ru


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: W state, Apache DOS

On 23/02/2010 11:16 AM, Igor wrote:
> Hello All,
>
> I would like to share some experience with you as regarded to one cause of an
> issue with Apache threads hanging in W state, causing a temporary DOS
> of HTTP server.
>
> The issue was tracked down to session_open function and PHP garbage collector.
> When you have a relatively busy server with more than 50K hosts hitting PHP pages
> and about 200K sessions in tmp folder the behavior leading to W
> state is as follows:
>
> * PHP calls session_open
> * garbage collector is started
> * session_open is locked till GC is finished
> * by some reason (not yet investigated) PHP's GC process takes a lot of time, and I mean a lot, a
> lot more than 20 minutes, during that time all the subsequent calls of session_open
> are locked and waiting till the GC is over
> * apache threads are consumed
> * apache reaches maxclients
> * server resources are consumed but no load is generated
> * Apche DOS, no server load is generated it looks like GC just hangs
>
> by some reasons PHP's GC is not affected by execution time
> restrictions that means that no matter how long the resource
> policies aren't applied to GC and the script will hang in session_open
> state almost indefinitely
>
> A simple work around to that problem would be something like:
>
> #find /tmp/* -name "sess_*" -type f -cmin +120 | xargs rm 2>/dev/null
>
> the goal is to keep minimum session files in GC tmp folder
>
> Another workaround would be to turn off PHP's GC completely and do it
> on your own.
>
> It is really a bad idea to "hook" system tasks to web calls, like GC
> does.
>
> Everyone knows that, why PHP is not applying some program that would
> handle GC independently, say as a crontab job, - I can't tell. It's just
> wrong by design.
>
> Hope this might shed some light on one cause of Apache's W state.
>
>
>

That would be mostly a flaw in mod_php. I've actually had a user report
this recently in the #httpd channel, too.

In any case, opened sessions in php are all-around very bad. Close them.

Frank.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org