Debugging seg faults in Apache mod_perl

I have two handlers, one a response handler and a second an output filter.
If either of these handlers run then they run fine for any number of
requests, if I have both of these handlers I get an untraceable seg fault
with the handlers, this segfault happens on the second request to that
particular child.

If the page is generated from:

* the file system,
* via mod_php or
* mod_rails I don't have a problem

If the page content is generated by mod_perl it works perfectly well for
the first request - but fails for subsequent requests.

[notice] child pid 14475 exit signal Segmentation fault (11)

This appears to be before the page handler executes...

If I turn off the output filter - everything is OK for all requests.

Any suggestions on how I debug this... it makes life so much easier if
I can handle all requests this way (and it seems to be the apache way)

James


--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

Re: Debugging seg faults in Apache mod_perl

Futher to this I now have a stack trace if that helps anyone point me in
the right direction:

#0 0x00007f17ee936fb1 in strncpy () from /lib/libc.so.6
#1 0x00007f17e808ecfc in modperl_perl_global_request_save () from
/usr/lib/apache2/modules/mod_perl.so
#2 0x00007f17e807dabc in modperl_response_handler_cgi () from
/usr/lib/apache2/modules/mod_perl.so
#3 0x00007f17ef7272d3 in ap_run_handler () from /usr/sbin/apache2
#4 0x00007f17ef72aa6f in ap_invoke_handler () from /usr/sbin/apache2
#5 0x00007f17ef7385de in ap_process_request () from /usr/sbin/apache2
#6 0x00007f17ef735418 in ?? () from /usr/sbin/apache2
#7 0x00007f17ef72eca3 in ap_run_process_connection () from
/usr/sbin/apache2
#8 0x00007f17ef73cf46 in ?? () from /usr/sbin/apache2
#9 0x00007f17ef73d276 in ?? () from /usr/sbin/apache2
#10 0x00007f17ef73ddad in ap_mpm_run () from /usr/sbin/apache2
#11 0x00007f17ef71360d in main () from /usr/sbin/apache2


On Mon, 17 Aug 2009, James Smith wrote:

>
> I have two handlers, one a response handler and a second an output filter.
> If either of these handlers run then they run fine for any number of
> requests, if I have both of these handlers I get an untraceable seg fault
> with the handlers, this segfault happens on the second request to that
> particular child.
>
> If the page is generated from:
>
> * the file system,
> * via mod_php or
> * mod_rails I don't have a problem
>
> If the page content is generated by mod_perl it works perfectly well for
> the first request - but fails for subsequent requests.
>
> [notice] child pid 14475 exit signal Segmentation fault (11)
>
> This appears to be before the page handler executes...
>
> If I turn off the output filter - everything is OK for all requests.
>
> Any suggestions on how I debug this... it makes life so much easier if
> I can handle all requests this way (and it seems to be the apache way)
>
> James
>
>
> --
> The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered in
> England with number 2742969, whose registered office is 215 Euston Road,
> London, NW1 2BE.


--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

Re: Debugging seg faults in Apache mod_perl

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig67D84E55BDCADF3859BE52FA
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 17/08/09 19:54 , James Smith wrote:
>=20
> Futher to this I now have a stack trace if that helps anyone point me i=
n
> the right direction:
>=20
> #0 0x00007f17ee936fb1 in strncpy () from /lib/libc.so.6
> #1 0x00007f17e808ecfc in modperl_perl_global_request_save () from
> /usr/lib/apache2/modules/mod_perl.so

That's hapenning in a fairly simple piece of code:

modperl_perl_global_svpv_save(pTHX_ modperl_perl_global_svpv_t *svpv)
{
svpv->cur =3D SvCUR(*svpv->sv);
strncpy(svpv->pv, SvPVX(*svpv->sv), sizeof(svpv->pv));
}

Looking at this, is there any chance you can get a mod_perl with
debugging symbols running this and post the content of :

*svpv
*(svpv->pv)
*(svpv->sv)

But looking at the guts, the only code path that will invoke this is
when saving/restoring the value of $/... Can you check to see what part
of your code alters that variable and to what ?

It might point to something interesting.

> #2 0x00007f17e807dabc in modperl_response_handler_cgi () from
> /usr/lib/apache2/modules/mod_perl.so
> #3 0x00007f17ef7272d3 in ap_run_handler () from /usr/sbin/apache2
> #4 0x00007f17ef72aa6f in ap_invoke_handler () from /usr/sbin/apache2
> #5 0x00007f17ef7385de in ap_process_request () from /usr/sbin/apache2
> #6 0x00007f17ef735418 in ?? () from /usr/sbin/apache2
> #7 0x00007f17ef72eca3 in ap_run_process_connection () from
> /usr/sbin/apache2
> #8 0x00007f17ef73cf46 in ?? () from /usr/sbin/apache2
> #9 0x00007f17ef73d276 in ?? () from /usr/sbin/apache2
> #10 0x00007f17ef73ddad in ap_mpm_run () from /usr/sbin/apache2
> #11 0x00007f17ef71360d in main () from /usr/sbin/apache2
>=20
>=20
> On Mon, 17 Aug 2009, James Smith wrote:
>=20
>>
>> I have two handlers, one a response handler and a second an output
>> filter.
>> If either of these handlers run then they run fine for any number of
>> requests, if I have both of these handlers I get an untraceable seg fa=
ult
>> with the handlers, this segfault happens on the second request to that=

>> particular child.
>>
>> If the page is generated from:
>>
>> * the file system,
>> * via mod_php or
>> * mod_rails I don't have a problem
>>
>> If the page content is generated by mod_perl it works perfectly well f=
or
>> the first request - but fails for subsequent requests.
>>
>> [notice] child pid 14475 exit signal Segmentation fault (11)
>>
>> This appears to be before the page handler executes...
>>
>> If I turn off the output filter - everything is OK for all requests.
>>
>> Any suggestions on how I debug this... it makes life so much easier if=

>> I can handle all requests this way (and it seems to be the apache way)=

>>
>> James
>>
>>
>> --=20
>> The Wellcome Trust Sanger Institute is operated by Genome Research
>> Limited, a charity registered in England with number 1021457 and a
>> company registered in England with number 2742969, whose registered
>> office is 215 Euston Road, London, NW1 2BE.
>=20
>=20


--=20
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\@(apache|cpan|ectoplasm)\.org/


--------------enig67D84E55BDCADF3859BE52FA
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFKigiKyzKhB4jDpaURAsggAJ9B59ERDEgNFPRFa6UDorCr8BE0KwCd GqiY
MXjsPPTTSusmeRrZlC9rcrw=
=1WPk
-----END PGP SIGNATURE-----

--------------enig67D84E55BDCADF3859BE52FA--

Re: Debugging seg faults in Apache mod_perl

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigC7C22532643D656EC630AD68
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 17/08/09 21:48 , Philippe M. Chiasson wrote:
> On 17/08/09 19:54 , James Smith wrote:
>>
>> Futher to this I now have a stack trace if that helps anyone point me =
in
>> the right direction:
>>
>> #0 0x00007f17ee936fb1 in strncpy () from /lib/libc.so.6
>> #1 0x00007f17e808ecfc in modperl_perl_global_request_save () from
>> /usr/lib/apache2/modules/mod_perl.so
>=20
> That's hapenning in a fairly simple piece of code:
>=20
> modperl_perl_global_svpv_save(pTHX_ modperl_perl_global_svpv_t *svpv)
> {
> svpv->cur =3D SvCUR(*svpv->sv);
> strncpy(svpv->pv, SvPVX(*svpv->sv), sizeof(svpv->pv));
> }
>=20
>[...]
>=20
> But looking at the guts, the only code path that will invoke this is
> when saving/restoring the value of $/... Can you check to see what par=
t
> of your code alters that variable and to what ?

Just had an idea, can't verify or test it right now, but I suspect this
might be caused by:

$/ =3D undef;

Could you have one of these lying around in your code somewhere ?

If so, can you try and change it to the usually recommended form:

local $/ =3D undef;

Hope this helps.

--=20
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\@(apache|cpan|ectoplasm)\.org/


--------------enigC7C22532643D656EC630AD68
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFKijloyzKhB4jDpaURAnFkAKDD4sjVJlQpNgHeT/2hBtIOQIeMQgCa Aibc
vP3oglndK9rMykAmiHDMATA=
=YzyC
-----END PGP SIGNATURE-----

--------------enigC7C22532643D656EC630AD68--

Re: Debugging seg faults in Apache mod_perl

Phillipe,

Thnx, that seems to have solved the problem - I'd been slurping in the
page template file with a $/.. it also explains why switching from
using perl-script to modperl as the handler type also resolved the
issue...

James

On Tue, 18 Aug 2009, Philippe M. Chiasson wrote:

> On 17/08/09 21:48 , Philippe M. Chiasson wrote:
>> On 17/08/09 19:54 , James Smith wrote:
>>>
>>> Futher to this I now have a stack trace if that helps anyone point me in
>>> the right direction:
>>>
>>> #0 0x00007f17ee936fb1 in strncpy () from /lib/libc.so.6
>>> #1 0x00007f17e808ecfc in modperl_perl_global_request_save () from
>>> /usr/lib/apache2/modules/mod_perl.so
>>
>> That's hapenning in a fairly simple piece of code:
>>
>> modperl_perl_global_svpv_save(pTHX_ modperl_perl_global_svpv_t *svpv)
>> {
>> svpv->cur = SvCUR(*svpv->sv);
>> strncpy(svpv->pv, SvPVX(*svpv->sv), sizeof(svpv->pv));
>> }
>>
>> [...]
>>
>> But looking at the guts, the only code path that will invoke this is
>> when saving/restoring the value of $/... Can you check to see what part
>> of your code alters that variable and to what ?
>
> Just had an idea, can't verify or test it right now, but I suspect this
> might be caused by:
>
> $/ = undef;
>
> Could you have one of these lying around in your code somewhere ?
>
> If so, can you try and change it to the usually recommended form:
>
> local $/ = undef;
>
> Hope this helps.
>
> --
> Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
> http://gozer.ectoplasm.org/ m/gozer\@(apache|cpan|ectoplasm)\.org/
>
>


--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

Re: Debugging seg faults in Apache mod_perl

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig6092D36E39E102000769F1E8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 18/08/09 02:30 , James Smith wrote:
>=20
> Phillipe,
>=20
> Thnx, that seems to have solved the problem - I'd been slurping in the
> page template file with a $/..=20

In theory, if you used local $/, it should work, maybe.

Can you try this small patch ?

Index: src/modules/perl/modperl_perl_global.c
==================== =====3D=
==================== =====3D=
=================3D
--- src/modules/perl/modperl_perl_global.c (revision 805561)
+++ src/modules/perl/modperl_perl_global.c (working copy)
@@ -372,7 +372,7 @@
modperl_perl_global_svpv_save(pTHX_ modperl_perl_global_svpv_t *svpv)
{
svpv->cur =3D SvCUR(*svpv->sv);
- strncpy(svpv->pv, SvPVX(*svpv->sv), sizeof(svpv->pv));
+ strncpy(svpv->pv, SvPV_nolen(*svpv->sv), sizeof(svpv->pv));
}

static void


> it also explains why switching from
> using perl-script to modperl as the handler type also resolved the issu=
e...

Yup, in perl-script mode, it does a lot of housekeeping behind the scenes=


--=20
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\@(apache|cpan|ectoplasm)\.org/


--------------enig6092D36E39E102000769F1E8
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFKiwopyzKhB4jDpaURAtgwAJ0R3iNxe8QQbJ/KO/plee5EJQSkSQCa A2Q7
cD93PyAryC5YDOJZ4519sHA=
=SOMy
-----END PGP SIGNATURE-----

--------------enig6092D36E39E102000769F1E8--