Apache 2.2.8 hangs on Solaris 9: Deadlock?

Hi all,

Okay, the title isn't very descriptive, but I haven't much to work with :-/

We're having a recurrent problem with Apache 2.2 on two Solaris 9 boxes:
it hangs, and becomes unresponsive. Connecting to the port 80, the
connection is established, but stays stuck, and send no data at all.

I think it started with version 2.2, though it's difficult to say when.

When the problem occurs, a simple restart is enough to get it to work
again (the children are killed, but the parent process is kept alive).

the only messages I find in the log are about «deadlock situations» (see
below), however, they happen quite a long time before the last working
request is logged. In the exemple below, the last log before restart is
at 14/Apr/2008:19:59:50 +0200. Nothing else there.

I only found the following post which might relate to it, except in my
case, Apache is built with the worker MPM, not prefork:

http://www.mail-archive.com/dev@httpd.apache.org/msg39632.ht ml

Any idea?

I'd like to keep the worker MPM if possible, AFAICT, it's more
lightweight and more adequate to our use.

TIA,

Laurent

[Fri Apr 11 18:53:07 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Fri Apr 11 19:04:58 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Sat Apr 12 02:36:16 2008] [error] [client 195.189.142.134] File does
not exist: /usr/local/apache-contenus/capri/favicon.ico, referer:
http://my.address/
[Sat Apr 12 10:27:59 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Mon Apr 14 14:41:46 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Mon Apr 14 16:16:11 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Mon Apr 14 18:42:38 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Mon Apr 14 19:48:16 2008] [emerg] (45)Deadlock situation
detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
process gracefully.
[Tue Apr 15 08:44:15 2008] [warn] child process 9042 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:15 2008] [warn] child process 7176 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:15 2008] [warn] child process 6424 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:15 2008] [warn] child process 8910 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:17 2008] [warn] child process 7176 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:19 2008] [warn] child process 7176 still did not exit,
sending a SIGTERM
[Tue Apr 15 08:44:21 2008] [error] child process 7176 still did not
exit, sending a SIGKILL
[Tue Apr 15 08:44:22 2008] [notice] SIGHUP received. Attempting to restart
[Tue Apr 15 08:44:24 2008] [notice] Digest: generating secret for digest
authentication ...
[Tue Apr 15 08:44:24 2008] [notice] Digest: done
[Tue Apr 15 08:44:25 2008] [notice] Apache/2.2.8 (Unix) mod_ssl/2.2.3
OpenSSL/0.9.8f DAV/2 PHP/4.4.7 configured -- resuming normal operations


--
/ Leader de Projet & Communauté | I'm working, but not speaking for
\ G11N http://fr.opensolaris.org | Bull Services http://www.bull.com
/ FOSUG http://guses.org |

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

On Tue, Apr 15, 2008 at 9:43 AM, Laurent Blume wrote:

> Any idea?

Two questions:
- Are you using any "third party" modules (ie, modules that didn't
come with the apache source itself)?
- Is any part of your apache install on an NFS share?

Krist

--
krist.vanbesien@gmail.com
krist@vanbesien.org
Bremgarten b. Bern, Switzerland
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

On Tue, Apr 15, 2008 at 3:43 AM, Laurent Blume wrote:

> [Fri Apr 11 18:53:07 2008] [emerg] (45)Deadlock situation
> detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown
> process gracefully.

http://marc.info/?t=118753748900003&r=1&w=2

If apachectl -V says fcntl is the default (APR_USE_FCNTL_SERIALIZE?)
you can hit this if you have two apache components using the default
type of lock.

Some are changeable:

AcceptMutex
SSLMutex

Some always use the default:

mod_ldap's cache lock

Some might be used by third party modules

??

--
Eric Covener
covener@gmail.com

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

Krist van Besien a écrit :
> Two questions:
> - Are you using any "third party" modules (ie, modules that didn't
> come with the apache source itself)?

Yes, PHP 4.4.8.

> - Is any part of your apache install on an NFS share?

No, all of it is local, including pages, logs, conf, etc.

Laurent

--
/ Leader de Projet & Communauté | I'm working, but not speaking for
\ G11N http://fr.opensolaris.org | Bull Services http://www.bull.com
/ FOSUG http://guses.org |

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

Eric Covener a écrit :
> http://marc.info/?t=118753748900003&r=1&w=2
>
> If apachectl -V says fcntl is the default (APR_USE_FCNTL_SERIALIZE?)
> you can hit this if you have two apache components using the default
> type of lock.

Yes, it's fnctl(), I understand it's the default on Solaris.

> Some are changeable:
>
> AcceptMutex
> SSLMutex
>
> Some always use the default:
>
> mod_ldap's cache lock
>
> Some might be used by third party modules

PHP, maybe?
The thread you pointed is very interesting, thanks a lot for it.
However, is there a way to be sure that those error messages are indeed
related to Apache hanging? Seems likely, but they occur at different
moments.

I'll look for how to change the mutexes I can change. Any other hint
will be welcome.

Thanks,

Laurent
--
/ Leader de Projet & Communauté | I'm working, but not speaking for
\ G11N http://fr.opensolaris.org | Bull Services http://www.bull.com
/ FOSUG http://guses.org |

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

On Tue, Apr 15, 2008 at 1:42 PM, Laurent Blume wrote:

> I'll look for how to change the mutexes I can change. Any other hint
> will be welcome.
You could indeed use:
AcceptMutex pthread

However, your problem might be PHP. Running PHP in a multithreaded
apache is not recommended. The reason is that PHP is "glue", that uses
a lot of other libraries on your system. It is hard to verify that all
the libraries used by PHP are thread safe.

Krist

--
krist.vanbesien@gmail.com
krist@vanbesien.org
Bremgarten b. Bern, Switzerland
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

Krist van Besien wrote:
> On Tue, Apr 15, 2008 at 1:42 PM, Laurent Blume wrote:
>
>> I'll look for how to change the mutexes I can change. Any other hint
>> will be welcome.
> You could indeed use:
> AcceptMutex pthread
>
> However, your problem might be PHP. Running PHP in a multithreaded
> apache is not recommended. The reason is that PHP is "glue", that uses
> a lot of other libraries on your system. It is hard to verify that all
> the libraries used by PHP are thread safe.

One option to seriously consider is keeping the worker MPM you desire, but
to configure PHP through fastcgi (e.g. you could try out the new
mod_proxy_fcgi, or use an appropriate module).

This allows your fcgi php processes to be happily single-thread-forked,
leaving them outside of the server's process space, and still enjoy worker
mpm for your non-generated content.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

William A. Rowe, Jr. a écrit :
> Krist van Besien wrote:
>> On Tue, Apr 15, 2008 at 1:42 PM, Laurent Blume
>> wrote:
>>
>>> I'll look for how to change the mutexes I can change. Any other hint
>>> will be welcome.
>> You could indeed use:
>> AcceptMutex pthread
>>
>> However, your problem might be PHP. Running PHP in a multithreaded
>> apache is not recommended. The reason is that PHP is "glue", that uses
>> a lot of other libraries on your system. It is hard to verify that all
>> the libraries used by PHP are thread safe.
>
> One option to seriously consider is keeping the worker MPM you desire, but
> to configure PHP through fastcgi (e.g. you could try out the new
> mod_proxy_fcgi, or use an appropriate module).
>
> This allows your fcgi php processes to be happily single-thread-forked,
> leaving them outside of the server's process space, and still enjoy worker
> mpm for your non-generated content.

Thanks, Krist, William. I knew about the PHP reluctancy regarding
threads, but it had worked for years, so I didn't think about it. It
definitely seems like a plausible culprit. I can at least easily try the
prefork version on one server, and check if I still get error messages
and hangups.

The fastcgi option seems to be an interesting possibility, I'll look
into it as well.

Thanks guys, I'm on my way of a clean prefork Apache rebuild and more
testing.

Laurent
--
/ Leader de Projet & Communauté | I'm working, but not speaking for
\ G11N http://fr.opensolaris.org | Bull Services http://www.bull.com
/ FOSUG http://guses.org |

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock?

On Tue, Apr 15, 2008 at 2:32 PM, Laurent Blume wrote:

> Thanks, Krist, William. I knew about the PHP reluctancy regarding
> threads, but it had worked for years, so I didn't think about it. It
> definitely seems like a plausible culprit. I can at least easily try the
> prefork version on one server, and check if I still get error messages
> and hangups.

It is possible to reliably run a mutlithreaded apache with PHP, but
there is no guarantee. It all appears to depend on the libraries you
use. Maybe a recent change in php code you run causes a library that
is not thread safe to be used?

Krist


--
krist.vanbesien@gmail.com
krist@vanbesien.org
Bremgarten b. Bern, Switzerland
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.2.8 hangs on Solaris 9: Deadlock? [SOLVED]

Guys, since I rebuilt Apache with prefork, the problem hasn't appeared,
so it seems the issue was definitely PHP + threads.

Thank you all who helped!

Laurent
--
/ Leader de Projet & Communauté | I'm working, but not speaking for
\ G11N http://fr.opensolaris.org | Bull Services http://www.bull.com
/ FOSUG http://guses.org |

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org