Apache hangs on reload

Hello:

I'm running PHP inside suexec with mod_fcgid and a couple wrapper =
scripts, and though it works properly, Apache now hangs whenever someone =
issues /etc/init.d/apache2 reload (which happens fairly frequently). =
Killing off all FCGI processes and the www-data Apache processes seems =
to let it continue working, but this shouldn't be happening.

Here's the setup for one of my vhosts (one other is set up similarly, =
and suexec is otherwise not used at all - mod_suexec is not enabled):

dr-xr-xr-x 2 www-webmail www-webmail 4096 2010-01-20 21:57 =
/var/lib/roundcube/fcgi-wrapper

-r-xr-xr-x 1 www-webmail www-webmail 95 2010-01-20 21:57 php-cgi
#!/bin/sh
export PHP_FCGI_CHILDREN=3D0
export PHP_FCGI_MAX_REQUESTS=3D4000
exec /usr/bin/php5-cgi

-rwxr-xr-x 1 root root 71 2010-01-20 21:34 =
php-cgi-suexec
#!/bin/sh
exec /usr/lib/apache2/suexec www-webmail www-webmail php-cgi

And here is the relevant line in the vhost configuration:

FCGIWrapper "/var/lib/roundcube/fcgi-wrapper/php-cgi-suexec" .php

Any thoughts or tips?

--=20
Jeremy R.
Owner, NovaWave Solutions
http://www.novawave.net/


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache hangs on reload

--------------------------------------------------
From: "Jeremy R."
Sent: 23 January, 2010 0:59
To:
Subject: [users@httpd] Apache hangs on reload

> Hello:
>
> I'm running PHP inside suexec with mod_fcgid and a couple wrapper scripts, and
> though it works properly, Apache now hangs whenever someone issues
> /etc/init.d/apache2 reload (which happens fairly frequently). Killing off all
> FCGI processes and the www-data Apache processes seems to let it continue
> working, but this shouldn't be happening.
>
> Here's the setup for one of my vhosts (one other is set up similarly, and
> suexec is otherwise not used at all - mod_suexec is not enabled):
>
> dr-xr-xr-x 2 www-webmail www-webmail 4096 2010-01-20 21:57
> /var/lib/roundcube/fcgi-wrapper
>
> -r-xr-xr-x 1 www-webmail www-webmail 95 2010-01-20 21:57 php-cgi
> #!/bin/sh
> export PHP_FCGI_CHILDREN=0
> export PHP_FCGI_MAX_REQUESTS=4000
> exec /usr/bin/php5-cgi
>
> -rwxr-xr-x 1 root root 71 2010-01-20 21:34 php-cgi-suexec
> #!/bin/sh
> exec /usr/lib/apache2/suexec www-webmail www-webmail php-cgi
>
> And here is the relevant line in the vhost configuration:
>
> FCGIWrapper "/var/lib/roundcube/fcgi-wrapper/php-cgi-suexec" .php
>
> Any thoughts or tips?
>
> --
> Jeremy R.
> Owner, NovaWave Solutions
> http://www.novawave.net/
>
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>

Jeremy,

Check your error logs.

Thanks,
Daniel


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache hangs on reload

Okay, so here are the salient lines from the error logs (I have two =
vhosts, webmail and wiki, running through suexec =96 if I put them =
through the regular FCGI process without suexec, I don't have this =
problem). Also, I should not that this problem does not occur on my =
testing VM, only on our production server (so I'm not quite sure what =
the problem is).

webmail-error.log:

[Wed Jan 20 23:38:17 2010] [emerg] mod_fcgid: server is restarted, 32760 =
must exit

wiki-error.log

[Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 22502 =
must exit
[Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: =
can't get lock, pid: 22502
[Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 16264 =
must exit
[Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: =
can't get lock, pid: 16264
[Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 22501 =
must exit
[Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: =
can't get lock, pid: 22501

error.log

[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/mediawiki/index.php(22557) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/mediawiki/index.php(22561) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/mediawiki/index.php(22562) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/roundcube/index.php(2380) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/roundcube/index.php(5653) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: process =
/var/lib/roundcube/index.php(6212) exit(shutting down), terminated by =
calling exit(), return code: 0
[Sat Jan 23 15:03:27 2010] [notice] mod_webauth: initialized (3.6.0) =
(Built by buildd@excelsior on 2008-05-02 22:03:49 UTC)
[Sat Jan 23 15:03:27 2010] [notice] Apache/2.2.9 (Debian) mod_ssl/2.2.9 =
OpenSSL/0.9.8g WebAuth/3.6.0 configured -- resuming normal operations
[Sat Jan 23 15:03:27 2010] [notice] mod_fcgid: call =
/var/lib/mediawiki/index.php with wrapper =
/var/lib/mediawiki/fcgi-wrapper/php-cgi-suexec
[Sat Jan 23 15:03:28 2010] [notice] mod_fcgid: call =
/var/lib/mediawiki/index.php with wrapper =
/var/lib/mediawiki/fcgi-wrapper/php-cgi-suexec
[Sat Jan 23 15:03:28 2010] [notice] mod_fcgid: call =
/var/lib/mediawiki/index.php with wrapper =
/var/lib/mediawiki/fcgi-wrapper/php-cgi-suexec

It looks like the emergency messages about being unable to get a lock on =
these processes is what's causing Apache to hang until I forcefully kill =
them.

On 2010-01-22, at 9:24 PM, Daniel Reinhardt wrote:

>=20
> --------------------------------------------------
> From: "Jeremy R."
> Sent: 23 January, 2010 0:59
> To:
> Subject: [users@httpd] Apache hangs on reload
>=20
>> Hello:
>>=20
>> I'm running PHP inside suexec with mod_fcgid and a couple wrapper =
scripts, and though it works properly, Apache now hangs whenever someone =
issues /etc/init.d/apache2 reload (which happens fairly frequently). =
Killing off all FCGI processes and the www-data Apache processes seems =
to let it continue working, but this shouldn't be happening.
>>=20
>> Here's the setup for one of my vhosts (one other is set up similarly, =
and suexec is otherwise not used at all - mod_suexec is not enabled):
>>=20
>> dr-xr-xr-x 2 www-webmail www-webmail 4096 2010-01-20 21:57 =
/var/lib/roundcube/fcgi-wrapper
>>=20
>> -r-xr-xr-x 1 www-webmail www-webmail 95 2010-01-20 21:57 php-cgi
>> #!/bin/sh
>> export PHP_FCGI_CHILDREN=3D0
>> export PHP_FCGI_MAX_REQUESTS=3D4000
>> exec /usr/bin/php5-cgi
>>=20
>> -rwxr-xr-x 1 root root 71 2010-01-20 21:34 =
php-cgi-suexec
>> #!/bin/sh
>> exec /usr/lib/apache2/suexec www-webmail www-webmail php-cgi
>>=20
>> And here is the relevant line in the vhost configuration:
>>=20
>> FCGIWrapper "/var/lib/roundcube/fcgi-wrapper/php-cgi-suexec" .php
>>=20
>> Any thoughts or tips?
>>=20
>> --=20
>> Jeremy R.
>> Owner, NovaWave Solutions
>> http://www.novawave.net/
>>=20
>>=20
>> ------------------------------------------------------------ ---------
>> The official User-To-User support forum of the Apache HTTP Server =
Project.
>> See for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>=20
>=20
> Jeremy,
>=20
> Check your error logs.
>=20
> Thanks,
> Daniel=20
>=20
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server =
Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>=20

--=20
Jeremy R.
Owner, NovaWave Solutions
http://www.novawave.net/


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache hangs on reload

On Sat, Jan 23, 2010 at 3:08 PM, Jeremy R. wrote:
> Okay, so here are the salient lines from the error logs (I have two vhost=
s, webmail and wiki, running through suexec =96 if I put them through the r=
egular FCGI process without suexec, I don't have this problem). Also, I sho=
uld not that this problem does not occur on my testing VM, only on our prod=
uction server (so I'm not quite sure what the problem is).
>
> webmail-error.log:
>
> [Wed Jan 20 23:38:17 2010] [emerg] mod_fcgid: server is restarted, 32760 =
must exit
>
> wiki-error.log
>
> [Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 22502 =
must exit
> [Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: can't=
get lock, pid: 22502
> [Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 16264 =
must exit
> [Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: can't=
get lock, pid: 16264
> [Sat Jan 23 15:03:25 2010] [emerg] mod_fcgid: server is restarted, 22501 =
must exit
> [Sat Jan 23 15:03:25 2010] [emerg] (22)Invalid argument: mod_fcgid: can't=
get lock, pid: 22501
>

> It looks like the emergency messages about being unable to get a lock on =
these processes is what's causing Apache to hang until I forcefully kill th=
em.

mod_fcgid tries to exit the httpd child process right after issuing
that message. I think it encounters the error because it is running
in a child process from the old generation (before restart) and the
lock was removed in the parent process.

Did you have to forcefully kill those same processes that said "can't
get lock"? Next time can you attach with gdb and get a backtrace?

Do something like this:

gdb /path/to/httpd 22501
....
(gdb) where
....
(gdb) quit

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache hangs on reload

On 2010-01-23, at 4:42 PM, Jeff Trawick wrote:

> mod_fcgid tries to exit the httpd child process right after issuing
> that message. I think it encounters the error because it is running
> in a child process from the old generation (before restart) and the
> lock was removed in the parent process.
>=20
> Did you have to forcefully kill those same processes that said "can't
> get lock"? Next time can you attach with gdb and get a backtrace?
>=20
> Do something like this:
>=20
> gdb /path/to/httpd 22501
> ...
> (gdb) where
> ...
> (gdb) quit


Doesn't seem to work (even when run as root):

# gdb /usr/sbin/apache2 14604
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later =

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show =
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)
Attaching to program: /usr/sbin/apache2, process 14604
ptrace: Operation not permitted.
/root/14604: No such file or directory.

Also, for some reason the message shows up in the main Apache log, and =
sometimes in the vhost's error log. If that helps.

--=20
Jeremy R.
Owner, NovaWave Solutions
http://www.novawave.net/


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org