Strange Apache behavior
am 26.03.2009 19:29:43 von binister--000e0cd213deca91d8046609ccd8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
I've run into a strange Apache/mod_perl issue recently. I have a mod_perl
application that has been running just fine for over 6 years. Recently
however I noticed that about 1 out of every 5 Apache restarts (we restart
nightly for various reasons) results in 500 responses. One (sometimes more)
restarts of the server fixes the problem.
I've tracked the issue down to the way Apache is handling each request. We
do a lot of process initialization in the ChildInitHandler. Normally when
Apache spawns a child process the ChildInitHandler runs and the request is
handled without a problem.
Recently I noticed that every so often the Apache parent process tries to
handle the request after a server restart. Our parent Apache process is very
lean and does not have any of the data structures/connections needed to
handle request. The parent process tries to handle every request when this
occurs. Children are not spawned. The only way to fix the problem is to
restart Apache.
Nothing in our environment has changed recently which makes this even more
curious.
Has anyone encountered such a problem before? I wasn't sure if I should post
this to the mod_perl or Apache mailing list. Details of our setup are posted
below.
Thanks for any help.
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=linux, osvers=2.6.9-67.0.15.elvm,
archname=i386-linux-thread-multi
uname='linux v20z-c4i386.home.local 2.6.9-67.0.15.elvm #1 wed may 7
13:46:12 edt 2008 i686 athlon i386 gnulinux '
config_args='-des -Doptimize=-O2 -g -pipe -m32 -march=i386
-mtune=pentium4 -Dversion=5.8.5 -Dmyhostname=localhost
-Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc.
-Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr
-Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles
-Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog
-Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly
-Dpager=/usr/bin/less -isr -Dinc_version_list=5.8.4 5.8.3 5.8.2 5.8.1 5.8.0'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
optimize='-O2 -g -pipe -m32 -march=i386 -mtune=pentium4',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing
-pipe -I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='3.4.6 20060404 (Red Hat 3.4.6-9)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.3.4.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.4'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE '
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Jun 12 2008 12:37:31
@INC:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/5.8.5
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl/5.8.4
/usr/lib/perl5/site_perl/5.8.3
/usr/lib/perl5/site_perl/5.8.2
/usr/lib/perl5/site_perl/5.8.1
/usr/lib/perl5/site_perl/5.8.0
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl/5.8.4
/usr/lib/perl5/vendor_perl/5.8.3
/usr/lib/perl5/vendor_perl/5.8.2
/usr/lib/perl5/vendor_perl/5.8.1
/usr/lib/perl5/vendor_perl/5.8.0
/usr/lib/perl5/vendor_perl
.
Perl version: 5.8.5
mod_perl version: 1.3
Apache version: 1.3
--000e0cd213deca91d8046609ccd8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
I've run into a strange Apache/mod_perl issue recently. I have a mod_pe=
rl application that has been running just fine for over 6 years. Recently h=
owever I noticed that about 1 out of every 5 Apache restarts (we restart ni=
ghtly for various reasons) results in 500 responses. One (sometimes more) r=
estarts of the server fixes the problem.
I've tracked the issue down to the way Apache is handling each requ=
est. We do a lot of process initialization in the ChildInitHandler. Normall=
y when Apache spawns a child process the ChildInitHandler runs and the requ=
est is handled without a problem.
Recently I noticed that every so often the Apache parent process tries =
to handle the request after a server restart. Our parent Apache process is =
very lean and does not have any of the data structures/connections needed t=
o handle request. The parent process tries to handle every request when thi=
s occurs. Children are not spawned. The only way to fix the problem is to r=
estart Apache.
Nothing in our environment has changed recently which makes this even m=
ore curious.
Has anyone encountered such a problem before? I wasn=
9;t sure if I should post this to the mod_perl or Apache mailing list. Deta=
ils of our setup are posted below.
Thanks for any help.
Summary of my perl5 (revision 5 version 8 =
subversion 5) configuration:
=A0 Platform:
=A0 osname=3Dlinux, =
osvers=3D2.6.9-67.0.15.elvm, archname=3Di386-linux-thread-multi
=
=A0 uname=3D'linux v20z-c4i386.home.local 2.6.9-67.0.15.elvm #1 wed may=
7 13:46:12 edt 2008 i686 athlon i386 gnulinux '
=A0 config_args=3D'-des -Doptimize=3D-O2 -g -pipe -m32 -march=3Di=
386 -mtune=3Dpentium4 -Dversion=3D5.8.5 -Dmyhostname=3Dlocalhost -Dperladmi=
n=3Droot@localhost -Dcc=3Dgcc -Dcf_by=3DRed Hat, Inc. -Dinstallprefix=3D/us=
r -Dprefix=3D/usr -Darchname=3Di386-linux -Dvendorprefix=3D/usr -Dsiteprefi=
x=3D/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid=
-Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=
=3D3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpage=
r=3D/usr/bin/less -isr -Dinc_version_list=3D5.8.4 5.8.3 5.8.2 5.8.1 5.8.0&#=
39;
=A0 hint=3Drecommended, useposix=3Dtrue, d_sigaction=3Ddefine
=A0=
usethreads=3Ddefine use5005threads=3Dundef useithreads=3Ddefine usem=
ultiplicity=3Ddefine
=A0 useperlio=3Ddefine d_sfio=3Dundef uselarg=
efiles=3Ddefine usesocks=3Dundef
=A0 use64bitint=3Dundef use64bita=
ll=3Dundef uselongdouble=3Dundef
=A0 usemymalloc=3Dn, bincompat5005=3Dundef
=A0 Compiler:
=
=A0 cc=3D'gcc', ccflags =3D'-D_REENTRANT -D_GNU_SOURCE -DDEBUGG=
ING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_=
FILE_OFFSET_BITS=3D64 -I/usr/include/gdbm',
=A0 optimize=3D'-O2 -g -pipe -m32 -march=3Di386 -mtune=3Dpentium4=
',
=A0 cppflags=3D'-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING =
-fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
t 3.4.6-9)', gccosandvers=3D''
=A0 intsize=3D4, longsize=3D4, ptrsize=3D4, doublesize=3D8, byteorder=
=3D1234
=A0 d_longlong=3Ddefine, longlongsize=3D8, d_longdbl=3Ddef=
ine, longdblsize=3D12
=A0 ivtype=3D'long', ivsize=3D4, nvt=
ype=3D'double', nvsize=3D8, Off_t=3D'off_t', lseeksize=3D8<=
br>
=A0 alignbytes=3D4, prototype=3Ddefine
=A0 Linker and Libraries:
=
=A0 libpth=3D/usr/local/lib /lib /usr/lib
=A0 libs=3D-lresol=
v -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
=A0 perllibs=3D-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
so=3Dso, useshrplib=3Dtrue, libperl=3Dlibperl.so
=A0 gnulibc_vers=
ion=3D'2.3.4'
=A0 Dynamic Linking:
=A0 dlsrc=3Ddl_dlopen.xs, dlext=3Dso, d_dlsymun=3Dundef, ccdlflags=3D=
'-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE =
9;
=A0 cccdlflags=3D'-fPIC', lddlflags=3D'-shared -L/u=
sr/local/lib'
Characteristics of this binary (from libperl):
=A0 Compile-time =
options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_=
CONTEXT
=A0 Built under linux
=A0 Compiled at Jun 12 2008 12:37:31
=A0 /usr/lib/perl5/5.8.5/i386-linux-thread-multi
=A0 /usr/li=
b/perl5/5.8.5
=A0 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread=
-multi
=A0 /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi<=
br> =A0 /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi
=A0 /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
=
=A0 /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi
=A0 /us=
r/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
=A0 /usr/lib/p=
erl5/site_perl/5.8.5
=A0 /usr/lib/perl5/site_perl/5.8.4
=A0 /usr/lib/perl5/site_p=
erl/5.8.3
=A0 /usr/lib/perl5/site_perl/5.8.2
=A0 /usr/lib=
/perl5/site_perl/5.8.1
=A0 /usr/lib/perl5/site_perl/5.8.0
=A0=
/usr/lib/perl5/site_perl
=A0 /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
=A0=
/usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi
=
=A0 /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi
=A0 /=
usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi
=A0 /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi
=A0=
/usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
=
=A0 /usr/lib/perl5/vendor_perl/5.8.5
=A0 /usr/lib/perl5/vendor_per=
l/5.8.4
=A0 /usr/lib/perl5/vendor_perl/5.8.3
=A0 /usr/lib/perl5/vendor_perl/5.8.2
=A0 /usr/lib/perl5/vend=
or_perl/5.8.1
=A0 /usr/lib/perl5/vendor_perl/5.8.0
=A0 /u=
sr/lib/perl5/vendor_perl
=A0 .
Perl version: 5.8.5
mod_perl =
version: 1.3
Apache version: 1.3
--000e0cd213deca91d8046609ccd8--