mod_perl2 + fork + DBI = Chaos

My application-from-hell is doing odd things probably related to the
above storms and culminating in a perfect storm.

Can anyone point me to resources outlining the best way to use the
combination in the subject line?

There seems to be many ways of accomplishing specifics but very little
generic methodologies which can be adapted.

What I'm trying to do is in a nutshell:

##########
my $object_with_DBH = new My::DataObject;

.... do stuff...

if ($pid = fork) {
... parent things ...
} elsif (defined $pid) {
... close STDhandles ...

my $data = $object_with_DBH->SQL('do SQL stuff with old connection');

exec 'things from $data';

CORE::exit(0);
}
##########

I've tried Stas' code but that's for MP1, and with $DBI->clone(), and
playing with "setsid" from POSIX and on... There seems to be no love
between mod_perl2, fork, and DBI.

On a side, but possibly related note, my browser is sending a POST
request which results in the FORK above and the parent then sends a 302
redirect with some status variables. This works fine, except between
the POST and the 302 GET is another POST to the same URI which results
in a 500 error, here's the log:

"POST /tosh/mailfile.pl HTTP/1.1" 302 282

"POST /tosh/mailfile.pl HTTP/1.1" 500 424

"GET /tosh/index.pl?message=Notice%20sent HTTP/1.1" 200 4309

The browser only seems to see the middle one, the 500. I'm certain this
is related to my poor forking above but I have no idea why. Are there
any tools for Firefox that will let me track every connection it's
making and why?

Thanks again for any help with my Frankenstein monster...

Tosh

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

RE: mod_perl2 + fork + DBI = Chaos

=20

>-----Message d'origine-----
>De : Tosh Cooey [mailto:tosh@1200group.com]=20
>Envoy=E9 : lundi, 25. janvier 2010 17:08
>=C0 : modperl@perl.apache.org
>Objet : mod_perl2 + fork + DBI =3D Chaos
>
>My application-from-hell is doing odd things probably related to the=20
>above storms and culminating in a perfect storm.
>
>Can anyone point me to resources outlining the best way to use the=20
>combination in the subject line?
>

See the DBI doc under "InactiveDestroy"; there are some words of advice =
about forks.

But anyway, as said in that doc, passing a dbh across a fork is not =
supported by all drivers.
In particular if your "..parent things.." also includes some access to =
the dbh, then you
might enter into nasty concurrency problems, and that might force you to =
change the architecture of your app ...

Good luck anyway!

Laurent Dami

Re: mod_perl2 + fork + DBI = Chaos

Sorry, I forgot to mention I checked InactiveDestroy as well, but since
my parent sends a 302 redirect to the browser right away and nothing
else is done in the parent I don't think that's the culprit.

Thanks for reminding me!

Tosh


Dami Laurent (PJ) wrote:
>
>
>> -----Message d'origine-----
>> De : Tosh Cooey [mailto:tosh@1200group.com]
>> Envoyé : lundi, 25. janvier 2010 17:08
>> À : modperl@perl.apache.org
>> Objet : mod_perl2 + fork + DBI = Chaos
>>
>> My application-from-hell is doing odd things probably related to the
>> above storms and culminating in a perfect storm.
>>
>> Can anyone point me to resources outlining the best way to use the
>> combination in the subject line?
>>
>
> See the DBI doc under "InactiveDestroy"; there are some words of advice about forks.
>
> But anyway, as said in that doc, passing a dbh across a fork is not supported by all drivers.
> In particular if your "..parent things.." also includes some access to the dbh, then you
> might enter into nasty concurrency problems, and that might force you to change the architecture of your app ...
>
> Good luck anyway!
>
> Laurent Dami
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

Have you been through these threads already?
The first one seems quite like what you're doing.

http://mail-archives.apache.org/mod_mbox/perl-modperl/200908 .mbox/%
3C59a07310908250820m1f789901g22eece5b2897cf3b@mail.gmail.com %3E
http://www.gossamer-threads.com/lists/modperl/modperl/100099

Is the time for the SQL inquiry so bad that you can't
just do the inquiry before the fork-and-exec bit?

Best of Luck,
cmac


On Jan 25, 2010, at 8:07 AM, Tosh Cooey wrote:

> My application-from-hell is doing odd things probably related to
> the above storms and culminating in a perfect storm.
>
> Can anyone point me to resources outlining the best way to use the
> combination in the subject line?
>
> There seems to be many ways of accomplishing specifics but very
> little generic methodologies which can be adapted.
>
> What I'm trying to do is in a nutshell:
>
> ##########
> my $object_with_DBH = new My::DataObject;
>
> ... do stuff...
>
> if ($pid = fork) {
> ... parent things ...
> } elsif (defined $pid) {
> ... close STDhandles ...
>
> my $data = $object_with_DBH->SQL('do SQL stuff with old
> connection');
>
> exec 'things from $data';
>
> CORE::exit(0);
> }
> ##########
>
> I've tried Stas' code but that's for MP1, and with $DBI->clone(),
> and playing with "setsid" from POSIX and on... There seems to be no
> love between mod_perl2, fork, and DBI.
>
> On a side, but possibly related note, my browser is sending a POST
> request which results in the FORK above and the parent then sends a
> 302 redirect with some status variables. This works fine, except
> between the POST and the 302 GET is another POST to the same URI
> which results in a 500 error, here's the log:
>
> "POST /tosh/mailfile.pl HTTP/1.1" 302 282
>
> "POST /tosh/mailfile.pl HTTP/1.1" 500 424
>
> "GET /tosh/index.pl?message=Notice%20sent HTTP/1.1" 200 4309
>
> The browser only seems to see the middle one, the 500. I'm certain
> this is related to my poor forking above but I have no idea why.
> Are there any tools for Firefox that will let me track every
> connection it's making and why?
>
> Thanks again for any help with my Frankenstein monster...
>
> Tosh
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

I wish I could do SQL before the fork, but I need to update tables with
results in the forks.

Just to make things stranger, after I make a fork and things don't
happen the way I expect them, subsequent calls to other programs running
under mod_perl are generating Segmentation Faults. This could be due to
the fact that the server is running in Europe and the SQL server in
North America. I would really love to hear that this can cause
segmentation faults because otherwise I have no idea where they are
coming from and in all my years using this same framework in mp1, mp2 or
just vanilla CGI I have never seen so many segfaults.

Bah...

Tosh


mackenna@animalhead.com wrote:
> Have you been through these threads already?
> The first one seems quite like what you're doing.
>
> http://mail-archives.apache.org/mod_mbox/perl-modperl/200908 .mbox/%3C59a07310908250820m1f789901g22eece5b2897cf3b@mail.gm ail.com%3E
>
> http://www.gossamer-threads.com/lists/modperl/modperl/100099
>
> Is the time for the SQL inquiry so bad that you can't
> just do the inquiry before the fork-and-exec bit?
>
> Best of Luck,
> cmac
>
>
> On Jan 25, 2010, at 8:07 AM, Tosh Cooey wrote:
>
>> My application-from-hell is doing odd things probably related to the
>> above storms and culminating in a perfect storm.
>>
>> Can anyone point me to resources outlining the best way to use the
>> combination in the subject line?
>>
>> There seems to be many ways of accomplishing specifics but very little
>> generic methodologies which can be adapted.
>>
>> What I'm trying to do is in a nutshell:
>>
>> ##########
>> my $object_with_DBH = new My::DataObject;
>>
>> ... do stuff...
>>
>> if ($pid = fork) {
>> ... parent things ...
>> } elsif (defined $pid) {
>> ... close STDhandles ...
>>
>> my $data = $object_with_DBH->SQL('do SQL stuff with old connection');
>>
>> exec 'things from $data';
>>
>> CORE::exit(0);
>> }
>> ##########
>>
>> I've tried Stas' code but that's for MP1, and with $DBI->clone(), and
>> playing with "setsid" from POSIX and on... There seems to be no love
>> between mod_perl2, fork, and DBI.
>>
>> On a side, but possibly related note, my browser is sending a POST
>> request which results in the FORK above and the parent then sends a
>> 302 redirect with some status variables. This works fine, except
>> between the POST and the 302 GET is another POST to the same URI which
>> results in a 500 error, here's the log:
>>
>> "POST /tosh/mailfile.pl HTTP/1.1" 302 282
>>
>> "POST /tosh/mailfile.pl HTTP/1.1" 500 424
>>
>> "GET /tosh/index.pl?message=Notice%20sent HTTP/1.1" 200 4309
>>
>> The browser only seems to see the middle one, the 500. I'm certain
>> this is related to my poor forking above but I have no idea why. Are
>> there any tools for Firefox that will let me track every connection
>> it's making and why?
>>
>> Thanks again for any help with my Frankenstein monster...
>>
>> Tosh
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

On 01/25/2010 02:12 PM, Tosh Cooey wrote:

> Just to make things stranger, after I make a fork and things don't
> happen the way I expect them, subsequent calls to other programs running
> under mod_perl are generating Segmentation Faults. This could be due to
> the fact that the server is running in Europe and the SQL server in
> North America. I would really love to hear that this can cause
> segmentation faults because otherwise I have no idea where they are
> coming from

This wouldn't cause segmentation faults. Network delays and timeouts
yes, but not segmentation faults.

But the overriding rule is that you can't use database handles opened
before a fork in the child processes after the fork. You need to close
those handles and reopen them.

--
Michael Peters
Plus Three, LP

Re: mod_perl2 + fork + DBI = Chaos

Just to follow up on this before anyone spends any time caring about it,
the seg faults could also be because one of the zombie apache2 processes
I've spawned like a Gremlin is answering the request and well...

So, if I've done the fork properly and pretend I have apache PIDs 1..10
running, and the child is 11 and it does it's thing and hits
CORE::exit(0) should that process then die? Or will I now see apache
processes 1..11?

What should I be looking for after a clean fork?

Tosh



Tosh Cooey wrote:
> I wish I could do SQL before the fork, but I need to update tables with
> results in the forks.
>
> Just to make things stranger, after I make a fork and things don't
> happen the way I expect them, subsequent calls to other programs running
> under mod_perl are generating Segmentation Faults. This could be due to
> the fact that the server is running in Europe and the SQL server in
> North America. I would really love to hear that this can cause
> segmentation faults because otherwise I have no idea where they are
> coming from and in all my years using this same framework in mp1, mp2 or
> just vanilla CGI I have never seen so many segfaults.
>
> Bah...
>
> Tosh
>
>
> mackenna@animalhead.com wrote:
>> Have you been through these threads already?
>> The first one seems quite like what you're doing.
>>
>> http://mail-archives.apache.org/mod_mbox/perl-modperl/200908 .mbox/%3C59a07310908250820m1f789901g22eece5b2897cf3b@mail.gm ail.com%3E
>>
>> http://www.gossamer-threads.com/lists/modperl/modperl/100099
>>
>> Is the time for the SQL inquiry so bad that you can't
>> just do the inquiry before the fork-and-exec bit?
>>
>> Best of Luck,
>> cmac
>>
>>
>> On Jan 25, 2010, at 8:07 AM, Tosh Cooey wrote:
>>
>>> My application-from-hell is doing odd things probably related to the
>>> above storms and culminating in a perfect storm.
>>>
>>> Can anyone point me to resources outlining the best way to use the
>>> combination in the subject line?
>>>
>>> There seems to be many ways of accomplishing specifics but very
>>> little generic methodologies which can be adapted.
>>>
>>> What I'm trying to do is in a nutshell:
>>>
>>> ##########
>>> my $object_with_DBH = new My::DataObject;
>>>
>>> ... do stuff...
>>>
>>> if ($pid = fork) {
>>> ... parent things ...
>>> } elsif (defined $pid) {
>>> ... close STDhandles ...
>>>
>>> my $data = $object_with_DBH->SQL('do SQL stuff with old connection');
>>>
>>> exec 'things from $data';
>>>
>>> CORE::exit(0);
>>> }
>>> ##########
>>>
>>> I've tried Stas' code but that's for MP1, and with $DBI->clone(), and
>>> playing with "setsid" from POSIX and on... There seems to be no love
>>> between mod_perl2, fork, and DBI.
>>>
>>> On a side, but possibly related note, my browser is sending a POST
>>> request which results in the FORK above and the parent then sends a
>>> 302 redirect with some status variables. This works fine, except
>>> between the POST and the 302 GET is another POST to the same URI
>>> which results in a 500 error, here's the log:
>>>
>>> "POST /tosh/mailfile.pl HTTP/1.1" 302 282
>>>
>>> "POST /tosh/mailfile.pl HTTP/1.1" 500 424
>>>
>>> "GET /tosh/index.pl?message=Notice%20sent HTTP/1.1" 200 4309
>>>
>>> The browser only seems to see the middle one, the 500. I'm certain
>>> this is related to my poor forking above but I have no idea why. Are
>>> there any tools for Firefox that will let me track every connection
>>> it's making and why?
>>>
>>> Thanks again for any help with my Frankenstein monster...
>>>
>>> Tosh
>>>
>>> --
>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

On Mon, Jan 25, 2010 at 2:41 PM, Tosh Cooey wrote:
> Just to follow up on this before anyone spends any time caring about it, the
> seg faults could also be because one of the zombie apache2 processes I've
> spawned like a Gremlin is answering the request and well...

I don't think so. That's not how apache works.

> So, if I've done the fork properly and pretend I have apache PIDs 1..10
> running, and the child is 11 and it does it's thing and hits CORE::exit(0)
> should that process then die?

Yes.

Here's some advice about using DBI with forking:
http://www.perlmonks.org/?node_id=752289

- Perrin

Re: mod_perl2 + fork + DBI = Chaos

Perrin Harkins wrote:
> On Mon, Jan 25, 2010 at 2:41 PM, Tosh Cooey wrote:
>> Just to follow up on this before anyone spends any time caring about it, the
>> seg faults could also be because one of the zombie apache2 processes I've
>> spawned like a Gremlin is answering the request and well...
>
> I don't think so. That's not how apache works.
>
>> So, if I've done the fork properly and pretend I have apache PIDs 1..10
>> running, and the child is 11 and it does it's thing and hits CORE::exit(0)
>> should that process then die?
>
> Yes.
>
> Here's some advice about using DBI with forking:
> http://www.perlmonks.org/?node_id=752289


Thanks Perrin, the forking, my child got a PID of 30033 and then
afterwards when I checked the processes (ps) for 30033 I see:

[apache2]

Is that what's supposed to happen?

That PM thread seems to indicate that I must disconnect every DBH, not
just the ones that I will use. That's correct? It seems
counterintuitive to disconnect all DBHs. Are you also suggesting the
use of Parallel::ForkManager for forks?

I hope at least I have the forking properly working so I can concentrate
on the DBI issue...

Tosh


--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey wrote:
> Thanks Perrin, the forking, my child got a PID of 30033 and then afterwards
> when I checked the processes (ps) for 30033 I see:
>
> [apache2]
>
> Is that what's supposed to happen?

After you call exit? No. It should be gone. That's a zombie process.

> That PM thread seems to indicate that I must disconnect every DBH, not just
> the ones that I will use.

Either that, or you need to set InactiveDestroy on all of them in the
child process. Otherwise, when the child exits, it messes up all of
them for the parent.

> Are you also suggesting the use of
> Parallel::ForkManager for forks?

No. The DBI stuff is the same with either.

- Perrin

Re: mod_perl2 + fork + DBI = Chaos

Ok, then maybe I need to supply some code here to try and get clarification:

mailfile.pl
###########
use strict;
....
use POSIX;

#gather needed modules and objects
my $fileOBJ = new MyOBJS::FILE($in->param('id'));
my $clientOBJ = new ...
my $userOBJ = new ...
# All OBJjects have a {DBH} property which is their DB handle
# I hear I have to disconnect these first, do I have to disconnect ALL?
$fileOBJ->{DBH}->disconnect;
$SIG{CHLD} = 'IGNORE';
my $pid;
if ($pid = fork) {
warn "Pid = $pid";
} elsif (defined $pid) {
close(STDOUT);
close(STDIN);
close(STDERR);

# chdir to /, stops the process from preventing an unmount
chdir '/' or die "Can't chdir to /: $!";
# dump our STDIN and STDOUT handles
open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
# redirect for logging
open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr: $!";
# Prevent locking to apache process
setsid or die "Can't start a new session: $!";

# Create file download link email
my $mailSTR = ...

# Send the mail possibly to many people
foreach my $person (@people) {
open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
print MAIL $mailSTR;
close(MAIL);
}

# Need to recreate the DBI connection on the $fileOBJ I hear
$fileOBJ = new MyOBJS::FILE($in->param('id'));

# Do some SQL to update the $fileOBJ status based on mailout
$fileOBJ->sql....

# create LOGGING Objects to log stuff
my $logOBJ = new ...
$logOBJ->sql...

CORE::exit(0);
}

print $in->redirect... # For the parent to redirect the browser

# Done.

Is there a glaring mistake in the above?

The parent does no more DB stuff, it just sends a redirect.

This runs under ModPerl::Registry.

I'd like to get at least one thing working tonight, either the forking
or the DBI, I'll be happy!

Tosh



Perrin Harkins wrote:
> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey wrote:
>> Thanks Perrin, the forking, my child got a PID of 30033 and then afterwards
>> when I checked the processes (ps) for 30033 I see:
>>
>> [apache2]
>>
>> Is that what's supposed to happen?
>
> After you call exit? No. It should be gone. That's a zombie process.
>
>> That PM thread seems to indicate that I must disconnect every DBH, not just
>> the ones that I will use.
>
> Either that, or you need to set InactiveDestroy on all of them in the
> child process. Otherwise, when the child exits, it messes up all of
> them for the parent.
>
>> Are you also suggesting the use of
>> Parallel::ForkManager for forks?
>
> No. The DBI stuff is the same with either.
>
> - Perrin
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

You made no comment on the links I sent you earlier today.
They had lots of good advice. Particularly the first one
suggested not forking the Apache process, but using an
ap(1) call to start a process to do the additional processing.

OK, the ap(1) alternative was a bit light on details.

How about the alternative offered by Perrin Hawkins in the
same thread, of using a cleanup handler to do the follow-up
processing rather than a forked process.

From p. 107 of "mod_per2 User's Guide":

$r->push_handlers(PerlCleanupHandler => \&cleanup);
> print $in->redirect... # to redirect the browser

Now cleanup (which receives $r as its operand) can do
whatever slow stuff you need to, can probably use DBI
without all the pain you have below, and can access the
request to find out what to do.

In some past context you may have learned how to get hold of
a $r to use in these calls, and hopefully you're no longer
scared of $r. But there does remain the question of whether
a ModPerl::Registry module can do such calls.

Hopefully someone who knows can chime in on this.

If not, for me it would be worth the editing of getting the
module out from under ModPerl::Registry and into the "native
mode" of SetHandler modperl.

Best of luck,
cmac


On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:

> Ok, then maybe I need to supply some code here to try and get
> clarification:
>
> mailfile.pl
> ###########
> use strict;
> ...
> use POSIX;
>
> #gather needed modules and objects
> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
> my $clientOBJ = new ...
> my $userOBJ = new ...
> # All OBJjects have a {DBH} property which is their DB handle
> # I hear I have to disconnect these first, do I have to disconnect
> ALL?
> $fileOBJ->{DBH}->disconnect;
> $SIG{CHLD} = 'IGNORE';
> my $pid;
> if ($pid = fork) {
> warn "Pid = $pid";
> } elsif (defined $pid) {
> close(STDOUT);
> close(STDIN);
> close(STDERR);
>
> # chdir to /, stops the process from preventing an unmount
> chdir '/' or die "Can't chdir to /: $!";
> # dump our STDIN and STDOUT handles
> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
> # redirect for logging
> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr: $!";
> # Prevent locking to apache process
> setsid or die "Can't start a new session: $!";
>
> # Create file download link email
> my $mailSTR = ...
>
> # Send the mail possibly to many people
> foreach my $person (@people) {
> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
> print MAIL $mailSTR;
> close(MAIL);
> }
>
> # Need to recreate the DBI connection on the $fileOBJ I hear
> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>
> # Do some SQL to update the $fileOBJ status based on mailout
> $fileOBJ->sql....
>
> # create LOGGING Objects to log stuff
> my $logOBJ = new ...
> $logOBJ->sql...
>
> CORE::exit(0);
> }
>
> print $in->redirect... # For the parent to redirect the browser
>
> # Done.
>
> Is there a glaring mistake in the above?
>
> The parent does no more DB stuff, it just sends a redirect.
>
> This runs under ModPerl::Registry.
>
> I'd like to get at least one thing working tonight, either the
> forking or the DBI, I'll be happy!
>
> Tosh
>
>
>
> Perrin Harkins wrote:
>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>> wrote:
>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>> afterwards
>>> when I checked the processes (ps) for 30033 I see:
>>>
>>> [apache2]
>>>
>>> Is that what's supposed to happen?
>> After you call exit? No. It should be gone. That's a zombie
>> process.
>>> That PM thread seems to indicate that I must disconnect every
>>> DBH, not just
>>> the ones that I will use.
>> Either that, or you need to set InactiveDestroy on all of them in the
>> child process. Otherwise, when the child exits, it messes up all of
>> them for the parent.
>>> Are you also suggesting the use of
>>> Parallel::ForkManager for forks?
>> No. The DBI stuff is the same with either.
>> - Perrin
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which you
say below) is that an abbreviation for something?

And Perrin saying "cleanup handler" is right up there with "prostate
exam" in my list of things to get into, both scare me!

Of course at some point a man needs to do both...

So... If this magic: $r->push_handlers(PerlCleanupHandler => \&cleanup);
is available in ModPerl::Registry context then I will attempt to force
all my forks into early retirement and work the problem out that way.

Unfortunately Google doesn't return an easy answer, anybody know this
before I spend all day tomorrow in my struggle?

Thank-you all again,

Tosh


mackenna@animalhead.com wrote:
> You made no comment on the links I sent you earlier today.
> They had lots of good advice. Particularly the first one
> suggested not forking the Apache process, but using an
> ap(1) call to start a process to do the additional processing.
>
> OK, the ap(1) alternative was a bit light on details.
>
> How about the alternative offered by Perrin Hawkins in the
> same thread, of using a cleanup handler to do the follow-up
> processing rather than a forked process.
>
> From p. 107 of "mod_per2 User's Guide":
>
> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>> print $in->redirect... # to redirect the browser
>
> Now cleanup (which receives $r as its operand) can do
> whatever slow stuff you need to, can probably use DBI
> without all the pain you have below, and can access the
> request to find out what to do.
>
> In some past context you may have learned how to get hold of
> a $r to use in these calls, and hopefully you're no longer
> scared of $r. But there does remain the question of whether
> a ModPerl::Registry module can do such calls.
>
> Hopefully someone who knows can chime in on this.
>
> If not, for me it would be worth the editing of getting the
> module out from under ModPerl::Registry and into the "native
> mode" of SetHandler modperl.
>
> Best of luck,
> cmac
>
>
> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>
>> Ok, then maybe I need to supply some code here to try and get
>> clarification:
>>
>> mailfile.pl
>> ###########
>> use strict;
>> ...
>> use POSIX;
>>
>> #gather needed modules and objects
>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>> my $clientOBJ = new ...
>> my $userOBJ = new ...
>> # All OBJjects have a {DBH} property which is their DB handle
>> # I hear I have to disconnect these first, do I have to disconnect ALL?
>> $fileOBJ->{DBH}->disconnect;
>> $SIG{CHLD} = 'IGNORE';
>> my $pid;
>> if ($pid = fork) {
>> warn "Pid = $pid";
>> } elsif (defined $pid) {
>> close(STDOUT);
>> close(STDIN);
>> close(STDERR);
>>
>> # chdir to /, stops the process from preventing an unmount
>> chdir '/' or die "Can't chdir to /: $!";
>> # dump our STDIN and STDOUT handles
>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>> # redirect for logging
>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr: $!";
>> # Prevent locking to apache process
>> setsid or die "Can't start a new session: $!";
>>
>> # Create file download link email
>> my $mailSTR = ...
>>
>> # Send the mail possibly to many people
>> foreach my $person (@people) {
>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>> print MAIL $mailSTR;
>> close(MAIL);
>> }
>>
>> # Need to recreate the DBI connection on the $fileOBJ I hear
>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>
>> # Do some SQL to update the $fileOBJ status based on mailout
>> $fileOBJ->sql....
>>
>> # create LOGGING Objects to log stuff
>> my $logOBJ = new ...
>> $logOBJ->sql...
>>
>> CORE::exit(0);
>> }
>>
>> print $in->redirect... # For the parent to redirect the browser
>>
>> # Done.
>>
>> Is there a glaring mistake in the above?
>>
>> The parent does no more DB stuff, it just sends a redirect.
>>
>> This runs under ModPerl::Registry.
>>
>> I'd like to get at least one thing working tonight, either the forking
>> or the DBI, I'll be happy!
>>
>> Tosh
>>
>>
>>
>> Perrin Harkins wrote:
>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey wrote:
>>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>>> afterwards
>>>> when I checked the processes (ps) for 30033 I see:
>>>>
>>>> [apache2]
>>>>
>>>> Is that what's supposed to happen?
>>> After you call exit? No. It should be gone. That's a zombie process.
>>>> That PM thread seems to indicate that I must disconnect every DBH,
>>>> not just
>>>> the ones that I will use.
>>> Either that, or you need to set InactiveDestroy on all of them in the
>>> child process. Otherwise, when the child exits, it messes up all of
>>> them for the parent.
>>>> Are you also suggesting the use of
>>>> Parallel::ForkManager for forks?
>>> No. The DBI stuff is the same with either.
>>> - Perrin
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

at(1) is a Unix command to start a process.
Assuming you're on a Unix/Linux box, type "man at" to get the story.

A cleanup handler is more pleasant than a prostate exam.

You can spend your life waiting for others. Just write a
routine called "cleanup" and have it do something like
make a log entry.

use Apache2::Const(); # defines OK
use Apache2::Log(); # defines warn
use Apache2::RequestUtil(); # defines push_handlers
....
sub cleanup {
my ($r) = @_;
$r->warn("cleanup was here");
return Apache2::Const::OK;
}
Then put a call like the one below in your ModPerl::Registry routine.
If the log entry shows up in error_log, you're on your way...

Good Luck,
cmac

P.S. Google doesn't index some sites well.
Look at http://perl.apache.org/docs/2.0/
particularly its API link.


On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:

> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which
> you say below) is that an abbreviation for something?
>
> And Perrin saying "cleanup handler" is right up there with
> "prostate exam" in my list of things to get into, both scare me!
>
> Of course at some point a man needs to do both...
>
> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
> \&cleanup); is available in ModPerl::Registry context then I will
> attempt to force all my forks into early retirement and work the
> problem out that way.
>
> Unfortunately Google doesn't return an easy answer, anybody know
> this before I spend all day tomorrow in my struggle?
>
> Thank-you all again,
>
> Tosh
>
>
> mackenna@animalhead.com wrote:
>> You made no comment on the links I sent you earlier today.
>> They had lots of good advice. Particularly the first one
>> suggested not forking the Apache process, but using an
>> ap(1) call to start a process to do the additional processing.
>> OK, the ap(1) alternative was a bit light on details.
>> How about the alternative offered by Perrin Hawkins in the
>> same thread, of using a cleanup handler to do the follow-up
>> processing rather than a forked process.
>> From p. 107 of "mod_per2 User's Guide":
>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>> print $in->redirect... # to redirect the browser
>> Now cleanup (which receives $r as its operand) can do
>> whatever slow stuff you need to, can probably use DBI
>> without all the pain you have below, and can access the
>> request to find out what to do.
>> In some past context you may have learned how to get hold of
>> a $r to use in these calls, and hopefully you're no longer
>> scared of $r. But there does remain the question of whether
>> a ModPerl::Registry module can do such calls.
>> Hopefully someone who knows can chime in on this.
>> If not, for me it would be worth the editing of getting the
>> module out from under ModPerl::Registry and into the "native
>> mode" of SetHandler modperl.
>> Best of luck,
>> cmac
>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>> Ok, then maybe I need to supply some code here to try and get
>>> clarification:
>>>
>>> mailfile.pl
>>> ###########
>>> use strict;
>>> ...
>>> use POSIX;
>>>
>>> #gather needed modules and objects
>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>> my $clientOBJ = new ...
>>> my $userOBJ = new ...
>>> # All OBJjects have a {DBH} property which is their DB handle
>>> # I hear I have to disconnect these first, do I have to
>>> disconnect ALL?
>>> $fileOBJ->{DBH}->disconnect;
>>> $SIG{CHLD} = 'IGNORE';
>>> my $pid;
>>> if ($pid = fork) {
>>> warn "Pid = $pid";
>>> } elsif (defined $pid) {
>>> close(STDOUT);
>>> close(STDIN);
>>> close(STDERR);
>>>
>>> # chdir to /, stops the process from preventing an unmount
>>> chdir '/' or die "Can't chdir to /: $!";
>>> # dump our STDIN and STDOUT handles
>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>>> # redirect for logging
>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/
>>> stderr: $!";
>>> # Prevent locking to apache process
>>> setsid or die "Can't start a new session: $!";
>>>
>>> # Create file download link email
>>> my $mailSTR = ...
>>>
>>> # Send the mail possibly to many people
>>> foreach my $person (@people) {
>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>> print MAIL $mailSTR;
>>> close(MAIL);
>>> }
>>>
>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>
>>> # Do some SQL to update the $fileOBJ status based on mailout
>>> $fileOBJ->sql....
>>>
>>> # create LOGGING Objects to log stuff
>>> my $logOBJ = new ...
>>> $logOBJ->sql...
>>>
>>> CORE::exit(0);
>>> }
>>>
>>> print $in->redirect... # For the parent to redirect the browser
>>>
>>> # Done.
>>>
>>> Is there a glaring mistake in the above?
>>>
>>> The parent does no more DB stuff, it just sends a redirect.
>>>
>>> This runs under ModPerl::Registry.
>>>
>>> I'd like to get at least one thing working tonight, either the
>>> forking or the DBI, I'll be happy!
>>>
>>> Tosh
>>>
>>>
>>>
>>> Perrin Harkins wrote:
>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>> wrote:
>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and
>>>>> then afterwards
>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>
>>>>> [apache2]
>>>>>
>>>>> Is that what's supposed to happen?
>>>> After you call exit? No. It should be gone. That's a zombie
>>>> process.
>>>>> That PM thread seems to indicate that I must disconnect every
>>>>> DBH, not just
>>>>> the ones that I will use.
>>>> Either that, or you need to set InactiveDestroy on all of them
>>>> in the
>>>> child process. Otherwise, when the child exits, it messes up
>>>> all of
>>>> them for the parent.
>>>>> Are you also suggesting the use of
>>>>> Parallel::ForkManager for forks?
>>>> No. The DBI stuff is the same with either.
>>>> - Perrin
>>>
>>> --
>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>> 1200group.com/
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

On a FYI level the following works:

#!/usr/bin/perl

use strict;
use Apache2::Const(); # defines OK
use Apache2::Log(); # defines warn
use Apache2::RequestUtil(); # defines push_handlers


my $r = shift;

$r->push_handlers(PerlCleanupHandler => \&cleanup);

print "Content-type: text/html\n\n";
foreach my $key (sort keys(%ENV)) {
print "$key = $ENV{$key}

";
}

sub cleanup {
my ($r) = @_;
$r->warn("cleanup was here");
return Apache2::Const::OK;
}

######

Looks like I may have to retire forks AND get a prostate exam all at the
same time...

Tosh


mackenna@animalhead.com wrote:
> at(1) is a Unix command to start a process.
> Assuming you're on a Unix/Linux box, type "man at" to get the story.
>
> A cleanup handler is more pleasant than a prostate exam.
>
> You can spend your life waiting for others. Just write a
> routine called "cleanup" and have it do something like
> make a log entry.
>
> use Apache2::Const(); # defines OK
> use Apache2::Log(); # defines warn
> use Apache2::RequestUtil(); # defines push_handlers
> ...
> sub cleanup {
> my ($r) = @_;
> $r->warn("cleanup was here");
> return Apache2::Const::OK;
> }
> Then put a call like the one below in your ModPerl::Registry routine.
> If the log entry shows up in error_log, you're on your way...
>
> Good Luck,
> cmac
>
> P.S. Google doesn't index some sites well.
> Look at http://perl.apache.org/docs/2.0/
> particularly its API link.
>
>
> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>
>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which
>> you say below) is that an abbreviation for something?
>>
>> And Perrin saying "cleanup handler" is right up there with "prostate
>> exam" in my list of things to get into, both scare me!
>>
>> Of course at some point a man needs to do both...
>>
>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>> \&cleanup); is available in ModPerl::Registry context then I will
>> attempt to force all my forks into early retirement and work the
>> problem out that way.
>>
>> Unfortunately Google doesn't return an easy answer, anybody know this
>> before I spend all day tomorrow in my struggle?
>>
>> Thank-you all again,
>>
>> Tosh
>>
>>
>> mackenna@animalhead.com wrote:
>>> You made no comment on the links I sent you earlier today.
>>> They had lots of good advice. Particularly the first one
>>> suggested not forking the Apache process, but using an
>>> ap(1) call to start a process to do the additional processing.
>>> OK, the ap(1) alternative was a bit light on details.
>>> How about the alternative offered by Perrin Hawkins in the
>>> same thread, of using a cleanup handler to do the follow-up
>>> processing rather than a forked process.
>>> From p. 107 of "mod_per2 User's Guide":
>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>> print $in->redirect... # to redirect the browser
>>> Now cleanup (which receives $r as its operand) can do
>>> whatever slow stuff you need to, can probably use DBI
>>> without all the pain you have below, and can access the
>>> request to find out what to do.
>>> In some past context you may have learned how to get hold of
>>> a $r to use in these calls, and hopefully you're no longer
>>> scared of $r. But there does remain the question of whether
>>> a ModPerl::Registry module can do such calls.
>>> Hopefully someone who knows can chime in on this.
>>> If not, for me it would be worth the editing of getting the
>>> module out from under ModPerl::Registry and into the "native
>>> mode" of SetHandler modperl.
>>> Best of luck,
>>> cmac
>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>> Ok, then maybe I need to supply some code here to try and get
>>>> clarification:
>>>>
>>>> mailfile.pl
>>>> ###########
>>>> use strict;
>>>> ...
>>>> use POSIX;
>>>>
>>>> #gather needed modules and objects
>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>> my $clientOBJ = new ...
>>>> my $userOBJ = new ...
>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>> # I hear I have to disconnect these first, do I have to disconnect ALL?
>>>> $fileOBJ->{DBH}->disconnect;
>>>> $SIG{CHLD} = 'IGNORE';
>>>> my $pid;
>>>> if ($pid = fork) {
>>>> warn "Pid = $pid";
>>>> } elsif (defined $pid) {
>>>> close(STDOUT);
>>>> close(STDIN);
>>>> close(STDERR);
>>>>
>>>> # chdir to /, stops the process from preventing an unmount
>>>> chdir '/' or die "Can't chdir to /: $!";
>>>> # dump our STDIN and STDOUT handles
>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>>>> # redirect for logging
>>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr:
>>>> $!";
>>>> # Prevent locking to apache process
>>>> setsid or die "Can't start a new session: $!";
>>>>
>>>> # Create file download link email
>>>> my $mailSTR = ...
>>>>
>>>> # Send the mail possibly to many people
>>>> foreach my $person (@people) {
>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>> print MAIL $mailSTR;
>>>> close(MAIL);
>>>> }
>>>>
>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>
>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>> $fileOBJ->sql....
>>>>
>>>> # create LOGGING Objects to log stuff
>>>> my $logOBJ = new ...
>>>> $logOBJ->sql...
>>>>
>>>> CORE::exit(0);
>>>> }
>>>>
>>>> print $in->redirect... # For the parent to redirect the browser
>>>>
>>>> # Done.
>>>>
>>>> Is there a glaring mistake in the above?
>>>>
>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>
>>>> This runs under ModPerl::Registry.
>>>>
>>>> I'd like to get at least one thing working tonight, either the
>>>> forking or the DBI, I'll be happy!
>>>>
>>>> Tosh
>>>>
>>>>
>>>>
>>>> Perrin Harkins wrote:
>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>> wrote:
>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>>>>> afterwards
>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>
>>>>>> [apache2]
>>>>>>
>>>>>> Is that what's supposed to happen?
>>>>> After you call exit? No. It should be gone. That's a zombie
>>>>> process.
>>>>>> That PM thread seems to indicate that I must disconnect every DBH,
>>>>>> not just
>>>>>> the ones that I will use.
>>>>> Either that, or you need to set InactiveDestroy on all of them in the
>>>>> child process. Otherwise, when the child exits, it messes up all of
>>>>> them for the parent.
>>>>>> Are you also suggesting the use of
>>>>>> Parallel::ForkManager for forks?
>>>>> No. The DBI stuff is the same with either.
>>>>> - Perrin
>>>>
>>>> --
>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

302 Redirect not working as expected with PerlCleanupHandler andFirefox under ModPerl::Registry

So this works almost perfectly... Almost because:

#!/usr/bin/perl

use strict;
use Apache2::Const(); # defines OK
use Apache2::Log(); # defines warn
use Apache2::RequestUtil(); # defines push_handlers

my $r = shift;
$r->push_handlers(PerlCleanupHandler => \&cleanup );

$r->headers_out->set(Location => 'http://...index.pl');
$r->status(Apache2::Const::REDIRECT);
return Apache2::Const::REDIRECT;

sub cleanup {
my ($r) = @_;
$r->warn("Starting cleanup");
foreach my $num (1..5) {
$r->warn("Number is $num");
sleep 2;
}
return Apache2::Const::OK;
}
######## test.pl


It seems if you take the above program and hit it with Firefox (3.5.7
and 3.6) it may take 10 seconds (5 x sleep 2) before Firefox does the
redirect. Safari 4.0.4 seems fine. curl works as well :)

I said "may" above because it's not consistent. If you launch Firefox
fresh and hit the above program it may redirect instantly, but then
subsequent hits will illustrate the delay. I'm also seeing varying
behaviour on a different server that has no Basic Auth, but always the
problem is there.

Can anyone else reproduce this?

Thank-you!

Tosh


mackenna@animalhead.com wrote:
> at(1) is a Unix command to start a process.
> Assuming you're on a Unix/Linux box, type "man at" to get the story.
>
> A cleanup handler is more pleasant than a prostate exam.
>
> You can spend your life waiting for others. Just write a
> routine called "cleanup" and have it do something like
> make a log entry.
>
> use Apache2::Const(); # defines OK
> use Apache2::Log(); # defines warn
> use Apache2::RequestUtil(); # defines push_handlers
> ...
> sub cleanup {
> my ($r) = @_;
> $r->warn("cleanup was here");
> return Apache2::Const::OK;
> }
> Then put a call like the one below in your ModPerl::Registry routine.
> If the log entry shows up in error_log, you're on your way...
>
> Good Luck,
> cmac
>
> P.S. Google doesn't index some sites well.
> Look at http://perl.apache.org/docs/2.0/
> particularly its API link.
>
>
> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>
>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which
>> you say below) is that an abbreviation for something?
>>
>> And Perrin saying "cleanup handler" is right up there with "prostate
>> exam" in my list of things to get into, both scare me!
>>
>> Of course at some point a man needs to do both...
>>
>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>> \&cleanup); is available in ModPerl::Registry context then I will
>> attempt to force all my forks into early retirement and work the
>> problem out that way.
>>
>> Unfortunately Google doesn't return an easy answer, anybody know this
>> before I spend all day tomorrow in my struggle?
>>
>> Thank-you all again,
>>
>> Tosh
>>
>>
>> mackenna@animalhead.com wrote:
>>> You made no comment on the links I sent you earlier today.
>>> They had lots of good advice. Particularly the first one
>>> suggested not forking the Apache process, but using an
>>> ap(1) call to start a process to do the additional processing.
>>> OK, the ap(1) alternative was a bit light on details.
>>> How about the alternative offered by Perrin Hawkins in the
>>> same thread, of using a cleanup handler to do the follow-up
>>> processing rather than a forked process.
>>> From p. 107 of "mod_per2 User's Guide":
>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>> print $in->redirect... # to redirect the browser
>>> Now cleanup (which receives $r as its operand) can do
>>> whatever slow stuff you need to, can probably use DBI
>>> without all the pain you have below, and can access the
>>> request to find out what to do.
>>> In some past context you may have learned how to get hold of
>>> a $r to use in these calls, and hopefully you're no longer
>>> scared of $r. But there does remain the question of whether
>>> a ModPerl::Registry module can do such calls.
>>> Hopefully someone who knows can chime in on this.
>>> If not, for me it would be worth the editing of getting the
>>> module out from under ModPerl::Registry and into the "native
>>> mode" of SetHandler modperl.
>>> Best of luck,
>>> cmac
>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>> Ok, then maybe I need to supply some code here to try and get
>>>> clarification:
>>>>
>>>> mailfile.pl
>>>> ###########
>>>> use strict;
>>>> ...
>>>> use POSIX;
>>>>
>>>> #gather needed modules and objects
>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>> my $clientOBJ = new ...
>>>> my $userOBJ = new ...
>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>> # I hear I have to disconnect these first, do I have to disconnect ALL?
>>>> $fileOBJ->{DBH}->disconnect;
>>>> $SIG{CHLD} = 'IGNORE';
>>>> my $pid;
>>>> if ($pid = fork) {
>>>> warn "Pid = $pid";
>>>> } elsif (defined $pid) {
>>>> close(STDOUT);
>>>> close(STDIN);
>>>> close(STDERR);
>>>>
>>>> # chdir to /, stops the process from preventing an unmount
>>>> chdir '/' or die "Can't chdir to /: $!";
>>>> # dump our STDIN and STDOUT handles
>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>>>> # redirect for logging
>>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr:
>>>> $!";
>>>> # Prevent locking to apache process
>>>> setsid or die "Can't start a new session: $!";
>>>>
>>>> # Create file download link email
>>>> my $mailSTR = ...
>>>>
>>>> # Send the mail possibly to many people
>>>> foreach my $person (@people) {
>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>> print MAIL $mailSTR;
>>>> close(MAIL);
>>>> }
>>>>
>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>
>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>> $fileOBJ->sql....
>>>>
>>>> # create LOGGING Objects to log stuff
>>>> my $logOBJ = new ...
>>>> $logOBJ->sql...
>>>>
>>>> CORE::exit(0);
>>>> }
>>>>
>>>> print $in->redirect... # For the parent to redirect the browser
>>>>
>>>> # Done.
>>>>
>>>> Is there a glaring mistake in the above?
>>>>
>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>
>>>> This runs under ModPerl::Registry.
>>>>
>>>> I'd like to get at least one thing working tonight, either the
>>>> forking or the DBI, I'll be happy!
>>>>
>>>> Tosh
>>>>
>>>>
>>>>
>>>> Perrin Harkins wrote:
>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>> wrote:
>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>>>>> afterwards
>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>
>>>>>> [apache2]
>>>>>>
>>>>>> Is that what's supposed to happen?
>>>>> After you call exit? No. It should be gone. That's a zombie
>>>>> process.
>>>>>> That PM thread seems to indicate that I must disconnect every DBH,
>>>>>> not just
>>>>>> the ones that I will use.
>>>>> Either that, or you need to set InactiveDestroy on all of them in the
>>>>> child process. Otherwise, when the child exits, it messes up all of
>>>>> them for the parent.
>>>>>> Are you also suggesting the use of
>>>>>> Parallel::ForkManager for forks?
>>>>> No. The DBI stuff is the same with either.
>>>>> - Perrin
>>>>
>>>> --
>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: 302 Redirect not working as expected with PerlCleanupHandler and

Caveat Lector:

Long Cleanups done inline on the Apache children can cause problems.

If you get a situation where the CleanUp takes to long OR you get
enough traffic to the page(s) which engage the CleanUp then you will
encounter a tipping point, and soon after your website will be almost
completely unavailable. This will occur because the Apache Children
aren't processing requests fast enough to handle the rate at which the
come in; because they are busy in CleanUp.

The reason I bring this up is encountering the failure is usually a
catastrophic event. The website is almost always entirely down, and
workarounds can be hard to come by.

This may or may not apply to you depending on your traffic
characteristics and how long your cleanup takes, BUT it is something
you should be aware of.

-wjt

Re: 302 Redirect not working as expected with PerlCleanupHandler and Firefox under ModPerl::Registry

The warning from William T. made me think to ask:

Does your site have "KeepAlive On" in httpd.conf?
(If not I can't think of anything to suggest...)

If so, try adding this as part of the redirect:

use Apache2::Connection();
use Apache2::RequestRec();
....
my $c = $r->connection();
$c->keepalive(Apache2::Const::CONN_CLOSE);

This will keep your process (which is about to do a "long"
cleanup) from automatically getting the redirected request
from the browser.

Hopefully the root httpd will know that this "redirecting"
child has not finished the complete cycle, and will launch
other children if needed to process the redirected request
plus any other requests.

Of course William is right that if lots of requests are
arriving that need such cleanup, and the cleanup really does
take a long time on average, you are likely to pile up
more children than your household income (I'm sorry I meant
to say your server :-) can support.

Good Luck,
cmac


On Jan 26, 2010, at 2:27 PM, Tosh Cooey wrote:

> So this works almost perfectly... Almost because:
>
> #!/usr/bin/perl
>
> use strict;
> use Apache2::Const(); # defines OK
> use Apache2::Log(); # defines warn
> use Apache2::RequestUtil(); # defines push_handlers
>
> my $r = shift;
> $r->push_handlers(PerlCleanupHandler => \&cleanup );
>
> $r->headers_out->set(Location => 'http://...index.pl');
> $r->status(Apache2::Const::REDIRECT);
> return Apache2::Const::REDIRECT;
>
> sub cleanup {
> my ($r) = @_;
> $r->warn("Starting cleanup");
> foreach my $num (1..5) {
> $r->warn("Number is $num");
> sleep 2;
> }
> return Apache2::Const::OK;
> }
> ######## test.pl
>
>
> It seems if you take the above program and hit it with Firefox
> (3.5.7 and 3.6) it may take 10 seconds (5 x sleep 2) before Firefox
> does the redirect. Safari 4.0.4 seems fine. curl works as well :)
>
> I said "may" above because it's not consistent. If you launch
> Firefox fresh and hit the above program it may redirect instantly,
> but then subsequent hits will illustrate the delay. I'm also
> seeing varying behaviour on a different server that has no Basic
> Auth, but always the problem is there.
>
> Can anyone else reproduce this?
>
> Thank-you!
>
> Tosh
>
>
> mackenna@animalhead.com wrote:
>> at(1) is a Unix command to start a process.
>> Assuming you're on a Unix/Linux box, type "man at" to get the story.
>> A cleanup handler is more pleasant than a prostate exam.
>> You can spend your life waiting for others. Just write a
>> routine called "cleanup" and have it do something like
>> make a log entry.
>> use Apache2::Const(); # defines OK
>> use Apache2::Log(); # defines warn
>> use Apache2::RequestUtil(); # defines push_handlers
>> ...
>> sub cleanup {
>> my ($r) = @_;
>> $r->warn("cleanup was here");
>> return Apache2::Const::OK;
>> }
>> Then put a call like the one below in your ModPerl::Registry routine.
>> If the log entry shows up in error_log, you're on your way...
>> Good Luck,
>> cmac
>> P.S. Google doesn't index some sites well.
>> Look at http://perl.apache.org/docs/2.0/
>> particularly its API link.
>> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1)
>>> which you say below) is that an abbreviation for something?
>>>
>>> And Perrin saying "cleanup handler" is right up there with
>>> "prostate exam" in my list of things to get into, both scare me!
>>>
>>> Of course at some point a man needs to do both...
>>>
>>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>>> \&cleanup); is available in ModPerl::Registry context then I will
>>> attempt to force all my forks into early retirement and work the
>>> problem out that way.
>>>
>>> Unfortunately Google doesn't return an easy answer, anybody know
>>> this before I spend all day tomorrow in my struggle?
>>>
>>> Thank-you all again,
>>>
>>> Tosh
>>>
>>>
>>> mackenna@animalhead.com wrote:
>>>> You made no comment on the links I sent you earlier today.
>>>> They had lots of good advice. Particularly the first one
>>>> suggested not forking the Apache process, but using an
>>>> ap(1) call to start a process to do the additional processing.
>>>> OK, the ap(1) alternative was a bit light on details.
>>>> How about the alternative offered by Perrin Hawkins in the
>>>> same thread, of using a cleanup handler to do the follow-up
>>>> processing rather than a forked process.
>>>> From p. 107 of "mod_per2 User's Guide":
>>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>>> print $in->redirect... # to redirect the browser
>>>> Now cleanup (which receives $r as its operand) can do
>>>> whatever slow stuff you need to, can probably use DBI
>>>> without all the pain you have below, and can access the
>>>> request to find out what to do.
>>>> In some past context you may have learned how to get hold of
>>>> a $r to use in these calls, and hopefully you're no longer
>>>> scared of $r. But there does remain the question of whether
>>>> a ModPerl::Registry module can do such calls.
>>>> Hopefully someone who knows can chime in on this.
>>>> If not, for me it would be worth the editing of getting the
>>>> module out from under ModPerl::Registry and into the "native
>>>> mode" of SetHandler modperl.
>>>> Best of luck,
>>>> cmac
>>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>>> Ok, then maybe I need to supply some code here to try and get
>>>>> clarification:
>>>>>
>>>>> mailfile.pl
>>>>> ###########
>>>>> use strict;
>>>>> ...
>>>>> use POSIX;
>>>>>
>>>>> #gather needed modules and objects
>>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>> my $clientOBJ = new ...
>>>>> my $userOBJ = new ...
>>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>>> # I hear I have to disconnect these first, do I have to
>>>>> disconnect ALL?
>>>>> $fileOBJ->{DBH}->disconnect;
>>>>> $SIG{CHLD} = 'IGNORE';
>>>>> my $pid;
>>>>> if ($pid = fork) {
>>>>> warn "Pid = $pid";
>>>>> } elsif (defined $pid) {
>>>>> close(STDOUT);
>>>>> close(STDIN);
>>>>> close(STDERR);
>>>>>
>>>>> # chdir to /, stops the process from preventing an unmount
>>>>> chdir '/' or die "Can't chdir to /: $!";
>>>>> # dump our STDIN and STDOUT handles
>>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null:
>>>>> $!";
>>>>> # redirect for logging
>>>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/
>>>>> stderr: $!";
>>>>> # Prevent locking to apache process
>>>>> setsid or die "Can't start a new session: $!";
>>>>>
>>>>> # Create file download link email
>>>>> my $mailSTR = ...
>>>>>
>>>>> # Send the mail possibly to many people
>>>>> foreach my $person (@people) {
>>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>>> print MAIL $mailSTR;
>>>>> close(MAIL);
>>>>> }
>>>>>
>>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>>
>>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>>> $fileOBJ->sql....
>>>>>
>>>>> # create LOGGING Objects to log stuff
>>>>> my $logOBJ = new ...
>>>>> $logOBJ->sql...
>>>>>
>>>>> CORE::exit(0);
>>>>> }
>>>>>
>>>>> print $in->redirect... # For the parent to redirect the browser
>>>>>
>>>>> # Done.
>>>>>
>>>>> Is there a glaring mistake in the above?
>>>>>
>>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>>
>>>>> This runs under ModPerl::Registry.
>>>>>
>>>>> I'd like to get at least one thing working tonight, either the
>>>>> forking or the DBI, I'll be happy!
>>>>>
>>>>> Tosh
>>>>>
>>>>>
>>>>>
>>>>> Perrin Harkins wrote:
>>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>>> wrote:
>>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and
>>>>>>> then afterwards
>>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>>
>>>>>>> [apache2]
>>>>>>>
>>>>>>> Is that what's supposed to happen?
>>>>>> After you call exit? No. It should be gone. That's a zombie
>>>>>> process.
>>>>>>> That PM thread seems to indicate that I must disconnect every
>>>>>>> DBH, not just
>>>>>>> the ones that I will use.
>>>>>> Either that, or you need to set InactiveDestroy on all of them
>>>>>> in the
>>>>>> child process. Otherwise, when the child exits, it messes up
>>>>>> all of
>>>>>> them for the parent.
>>>>>>> Are you also suggesting the use of
>>>>>>> Parallel::ForkManager for forks?
>>>>>> No. The DBI stuff is the same with either.
>>>>>> - Perrin
>>>>>
>>>>> --
>>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>>>> 1200group.com/
>>>
>>> --
>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>> 1200group.com/
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: 302 Redirect not working as expected with PerlCleanupHandlerand Firefox under ModPerl::Registry

The good news is that Mr. Mackenna got it! If I set "KeepAlive Off" in
apache2.conf then it all works fine. Below is a functioning long
process thingy which works with "KeepAlive On" and Firefox. I just hope
it works with MSIE ...


#!/usr/bin/perl

use strict;
use Apache2::Const -compile => qw(:conn_keepalive);
use Apache2::Log(); # defines warn
use Apache2::RequestUtil(); # defines push_handlers
use Apache2::Connection();
use Apache2::RequestRec();

my $r = shift;
my $c = $r->connection();
$c->keepalive(Apache2::Const::CONN_CLOSE);
$r->push_handlers(PerlCleanupHandler => \&cleanup );

$r->err_headers_out->set(Location => 'http://...index.pl');
$r->status(Apache2::Const::REDIRECT);
return Apache2::Const::REDIRECT;

sub cleanup {
my ($r) = @_;
$r->warn("Starting cleanup");
foreach my $num (1..5) {
$r->warn("Number is $num");
sleep 2;
}
return Apache2::Const::OK;
}
####

So, thanks again, and as for the warning from William T., well my
spin-off processes are maybe 10s-10m long and not a driving feature, but
I will have to keep my eye on them.

Tosh


mackenna@animalhead.com wrote:
> The warning from William T. made me think to ask:
>
> Does your site have "KeepAlive On" in httpd.conf?
> (If not I can't think of anything to suggest...)
>
> If so, try adding this as part of the redirect:
>
> use Apache2::Connection();
> use Apache2::RequestRec();
> ...
> my $c = $r->connection();
> $c->keepalive(Apache2::Const::CONN_CLOSE);
>
> This will keep your process (which is about to do a "long"
> cleanup) from automatically getting the redirected request
> from the browser.
>
> Hopefully the root httpd will know that this "redirecting"
> child has not finished the complete cycle, and will launch
> other children if needed to process the redirected request
> plus any other requests.
>
> Of course William is right that if lots of requests are
> arriving that need such cleanup, and the cleanup really does
> take a long time on average, you are likely to pile up
> more children than your household income (I'm sorry I meant
> to say your server :-) can support.
>
> Good Luck,
> cmac
>
>
> On Jan 26, 2010, at 2:27 PM, Tosh Cooey wrote:
>
>> So this works almost perfectly... Almost because:
>>
>> #!/usr/bin/perl
>>
>> use strict;
>> use Apache2::Const(); # defines OK
>> use Apache2::Log(); # defines warn
>> use Apache2::RequestUtil(); # defines push_handlers
>>
>> my $r = shift;
>> $r->push_handlers(PerlCleanupHandler => \&cleanup );
>>
>> $r->headers_out->set(Location => 'http://...index.pl');
>> $r->status(Apache2::Const::REDIRECT);
>> return Apache2::Const::REDIRECT;
>>
>> sub cleanup {
>> my ($r) = @_;
>> $r->warn("Starting cleanup");
>> foreach my $num (1..5) {
>> $r->warn("Number is $num");
>> sleep 2;
>> }
>> return Apache2::Const::OK;
>> }
>> ######## test.pl
>>
>>
>> It seems if you take the above program and hit it with Firefox (3.5.7
>> and 3.6) it may take 10 seconds (5 x sleep 2) before Firefox does the
>> redirect. Safari 4.0.4 seems fine. curl works as well :)
>>
>> I said "may" above because it's not consistent. If you launch Firefox
>> fresh and hit the above program it may redirect instantly, but then
>> subsequent hits will illustrate the delay. I'm also seeing varying
>> behaviour on a different server that has no Basic Auth, but always the
>> problem is there.
>>
>> Can anyone else reproduce this?
>>
>> Thank-you!
>>
>> Tosh
>>
>>
>> mackenna@animalhead.com wrote:
>>> at(1) is a Unix command to start a process.
>>> Assuming you're on a Unix/Linux box, type "man at" to get the story.
>>> A cleanup handler is more pleasant than a prostate exam.
>>> You can spend your life waiting for others. Just write a
>>> routine called "cleanup" and have it do something like
>>> make a log entry.
>>> use Apache2::Const(); # defines OK
>>> use Apache2::Log(); # defines warn
>>> use Apache2::RequestUtil(); # defines push_handlers
>>> ...
>>> sub cleanup {
>>> my ($r) = @_;
>>> $r->warn("cleanup was here");
>>> return Apache2::Const::OK;
>>> }
>>> Then put a call like the one below in your ModPerl::Registry routine.
>>> If the log entry shows up in error_log, you're on your way...
>>> Good Luck,
>>> cmac
>>> P.S. Google doesn't index some sites well.
>>> Look at http://perl.apache.org/docs/2.0/
>>> particularly its API link.
>>> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>>>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which
>>>> you say below) is that an abbreviation for something?
>>>>
>>>> And Perrin saying "cleanup handler" is right up there with "prostate
>>>> exam" in my list of things to get into, both scare me!
>>>>
>>>> Of course at some point a man needs to do both...
>>>>
>>>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>>>> \&cleanup); is available in ModPerl::Registry context then I will
>>>> attempt to force all my forks into early retirement and work the
>>>> problem out that way.
>>>>
>>>> Unfortunately Google doesn't return an easy answer, anybody know
>>>> this before I spend all day tomorrow in my struggle?
>>>>
>>>> Thank-you all again,
>>>>
>>>> Tosh
>>>>
>>>>
>>>> mackenna@animalhead.com wrote:
>>>>> You made no comment on the links I sent you earlier today.
>>>>> They had lots of good advice. Particularly the first one
>>>>> suggested not forking the Apache process, but using an
>>>>> ap(1) call to start a process to do the additional processing.
>>>>> OK, the ap(1) alternative was a bit light on details.
>>>>> How about the alternative offered by Perrin Hawkins in the
>>>>> same thread, of using a cleanup handler to do the follow-up
>>>>> processing rather than a forked process.
>>>>> From p. 107 of "mod_per2 User's Guide":
>>>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>>>> print $in->redirect... # to redirect the browser
>>>>> Now cleanup (which receives $r as its operand) can do
>>>>> whatever slow stuff you need to, can probably use DBI
>>>>> without all the pain you have below, and can access the
>>>>> request to find out what to do.
>>>>> In some past context you may have learned how to get hold of
>>>>> a $r to use in these calls, and hopefully you're no longer
>>>>> scared of $r. But there does remain the question of whether
>>>>> a ModPerl::Registry module can do such calls.
>>>>> Hopefully someone who knows can chime in on this.
>>>>> If not, for me it would be worth the editing of getting the
>>>>> module out from under ModPerl::Registry and into the "native
>>>>> mode" of SetHandler modperl.
>>>>> Best of luck,
>>>>> cmac
>>>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>>>> Ok, then maybe I need to supply some code here to try and get
>>>>>> clarification:
>>>>>>
>>>>>> mailfile.pl
>>>>>> ###########
>>>>>> use strict;
>>>>>> ...
>>>>>> use POSIX;
>>>>>>
>>>>>> #gather needed modules and objects
>>>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>>> my $clientOBJ = new ...
>>>>>> my $userOBJ = new ...
>>>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>>>> # I hear I have to disconnect these first, do I have to disconnect
>>>>>> ALL?
>>>>>> $fileOBJ->{DBH}->disconnect;
>>>>>> $SIG{CHLD} = 'IGNORE';
>>>>>> my $pid;
>>>>>> if ($pid = fork) {
>>>>>> warn "Pid = $pid";
>>>>>> } elsif (defined $pid) {
>>>>>> close(STDOUT);
>>>>>> close(STDIN);
>>>>>> close(STDERR);
>>>>>>
>>>>>> # chdir to /, stops the process from preventing an unmount
>>>>>> chdir '/' or die "Can't chdir to /: $!";
>>>>>> # dump our STDIN and STDOUT handles
>>>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>>>>>> # redirect for logging
>>>>>> open STDERR, '>/tmp/stderr' or die "Can't write to
>>>>>> /tmp/stderr: $!";
>>>>>> # Prevent locking to apache process
>>>>>> setsid or die "Can't start a new session: $!";
>>>>>>
>>>>>> # Create file download link email
>>>>>> my $mailSTR = ...
>>>>>>
>>>>>> # Send the mail possibly to many people
>>>>>> foreach my $person (@people) {
>>>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>>>> print MAIL $mailSTR;
>>>>>> close(MAIL);
>>>>>> }
>>>>>>
>>>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>>>
>>>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>>>> $fileOBJ->sql....
>>>>>>
>>>>>> # create LOGGING Objects to log stuff
>>>>>> my $logOBJ = new ...
>>>>>> $logOBJ->sql...
>>>>>>
>>>>>> CORE::exit(0);
>>>>>> }
>>>>>>
>>>>>> print $in->redirect... # For the parent to redirect the browser
>>>>>>
>>>>>> # Done.
>>>>>>
>>>>>> Is there a glaring mistake in the above?
>>>>>>
>>>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>>>
>>>>>> This runs under ModPerl::Registry.
>>>>>>
>>>>>> I'd like to get at least one thing working tonight, either the
>>>>>> forking or the DBI, I'll be happy!
>>>>>>
>>>>>> Tosh
>>>>>>
>>>>>>
>>>>>>
>>>>>> Perrin Harkins wrote:
>>>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>>>> wrote:
>>>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>>>>>>> afterwards
>>>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>>>
>>>>>>>> [apache2]
>>>>>>>>
>>>>>>>> Is that what's supposed to happen?
>>>>>>> After you call exit? No. It should be gone. That's a zombie
>>>>>>> process.
>>>>>>>> That PM thread seems to indicate that I must disconnect every
>>>>>>>> DBH, not just
>>>>>>>> the ones that I will use.
>>>>>>> Either that, or you need to set InactiveDestroy on all of them in
>>>>>>> the
>>>>>>> child process. Otherwise, when the child exits, it messes up all of
>>>>>>> them for the parent.
>>>>>>>> Are you also suggesting the use of
>>>>>>>> Parallel::ForkManager for forks?
>>>>>>> No. The DBI stuff is the same with either.
>>>>>>> - Perrin
>>>>>>
>>>>>> --
>>>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>>>
>>>> --
>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: 302 Redirect not working as expected with PerlCleanupHandlerand Firefox under ModPerl::Registry

Would this problem be any different in a normal CGI context with the
program doing forks? I don't imagine it would be, which is why I see
the ultimate wisdom in spawning an external program to handle
long-running tasks, or just cron something.

Oh well, live and learn.

Tosh


William T wrote:
> Caveat Lector:
>
> Long Cleanups done inline on the Apache children can cause problems.
>
> If you get a situation where the CleanUp takes to long OR you get
> enough traffic to the page(s) which engage the CleanUp then you will
> encounter a tipping point, and soon after your website will be almost
> completely unavailable. This will occur because the Apache Children
> aren't processing requests fast enough to handle the rate at which the
> come in; because they are busy in CleanUp.
>
> The reason I bring this up is encountering the failure is usually a
> catastrophic event. The website is almost always entirely down, and
> workarounds can be hard to come by.
>
> This may or may not apply to you depending on your traffic
> characteristics and how long your cleanup takes, BUT it is something
> you should be aware of.
>
> -wjt
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

Just to tie this thread up...

In case anyone else is ever in the same situation I would like to tell
them that mod_perl(1|2) + fork = bad idea, and don't even THINK about
throwing DBI into the mix.

For me in the future, if I have a VERY long task I will externalize it,
and if I have brief but long tasks then the following worked great for me:

#!/usr/bin/perl

use strict;
use Apache2::Const -compile => qw(:conn_keepalive);
use Apache2::Log(); # defines warn
use Apache2::RequestUtil(); # defines push_handlers
use Apache2::Connection();
use Apache2::RequestRec();

my $r = shift;
my $c = $r->connection();
$c->keepalive(Apache2::Const::CONN_CLOSE);
$r->push_handlers(PerlCleanupHandler => \&cleanup );

$r->err_headers_out->set(Location => 'http://...index.pl);
$r->status(Apache2::Const::REDIRECT);
return Apache2::Const::REDIRECT;

sub cleanup { # long process
my ($r) = @_;
$r->warn("Starting cleanup");
foreach my $num (1..5) {
$r->warn("Number is $num");
sleep 2;
}
return Apache2::Const::OK;
}

####

YMMV.

Tosh



mackenna@animalhead.com wrote:
> at(1) is a Unix command to start a process.
> Assuming you're on a Unix/Linux box, type "man at" to get the story.
>
> A cleanup handler is more pleasant than a prostate exam.
>
> You can spend your life waiting for others. Just write a
> routine called "cleanup" and have it do something like
> make a log entry.
>
> use Apache2::Const(); # defines OK
> use Apache2::Log(); # defines warn
> use Apache2::RequestUtil(); # defines push_handlers
> ...
> sub cleanup {
> my ($r) = @_;
> $r->warn("cleanup was here");
> return Apache2::Const::OK;
> }
> Then put a call like the one below in your ModPerl::Registry routine.
> If the log entry shows up in error_log, you're on your way...
>
> Good Luck,
> cmac
>
> P.S. Google doesn't index some sites well.
> Look at http://perl.apache.org/docs/2.0/
> particularly its API link.
>
>
> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>
>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1) which
>> you say below) is that an abbreviation for something?
>>
>> And Perrin saying "cleanup handler" is right up there with "prostate
>> exam" in my list of things to get into, both scare me!
>>
>> Of course at some point a man needs to do both...
>>
>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>> \&cleanup); is available in ModPerl::Registry context then I will
>> attempt to force all my forks into early retirement and work the
>> problem out that way.
>>
>> Unfortunately Google doesn't return an easy answer, anybody know this
>> before I spend all day tomorrow in my struggle?
>>
>> Thank-you all again,
>>
>> Tosh
>>
>>
>> mackenna@animalhead.com wrote:
>>> You made no comment on the links I sent you earlier today.
>>> They had lots of good advice. Particularly the first one
>>> suggested not forking the Apache process, but using an
>>> ap(1) call to start a process to do the additional processing.
>>> OK, the ap(1) alternative was a bit light on details.
>>> How about the alternative offered by Perrin Hawkins in the
>>> same thread, of using a cleanup handler to do the follow-up
>>> processing rather than a forked process.
>>> From p. 107 of "mod_per2 User's Guide":
>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>> print $in->redirect... # to redirect the browser
>>> Now cleanup (which receives $r as its operand) can do
>>> whatever slow stuff you need to, can probably use DBI
>>> without all the pain you have below, and can access the
>>> request to find out what to do.
>>> In some past context you may have learned how to get hold of
>>> a $r to use in these calls, and hopefully you're no longer
>>> scared of $r. But there does remain the question of whether
>>> a ModPerl::Registry module can do such calls.
>>> Hopefully someone who knows can chime in on this.
>>> If not, for me it would be worth the editing of getting the
>>> module out from under ModPerl::Registry and into the "native
>>> mode" of SetHandler modperl.
>>> Best of luck,
>>> cmac
>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>> Ok, then maybe I need to supply some code here to try and get
>>>> clarification:
>>>>
>>>> mailfile.pl
>>>> ###########
>>>> use strict;
>>>> ...
>>>> use POSIX;
>>>>
>>>> #gather needed modules and objects
>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>> my $clientOBJ = new ...
>>>> my $userOBJ = new ...
>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>> # I hear I have to disconnect these first, do I have to disconnect ALL?
>>>> $fileOBJ->{DBH}->disconnect;
>>>> $SIG{CHLD} = 'IGNORE';
>>>> my $pid;
>>>> if ($pid = fork) {
>>>> warn "Pid = $pid";
>>>> } elsif (defined $pid) {
>>>> close(STDOUT);
>>>> close(STDIN);
>>>> close(STDERR);
>>>>
>>>> # chdir to /, stops the process from preventing an unmount
>>>> chdir '/' or die "Can't chdir to /: $!";
>>>> # dump our STDIN and STDOUT handles
>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
>>>> # redirect for logging
>>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/stderr:
>>>> $!";
>>>> # Prevent locking to apache process
>>>> setsid or die "Can't start a new session: $!";
>>>>
>>>> # Create file download link email
>>>> my $mailSTR = ...
>>>>
>>>> # Send the mail possibly to many people
>>>> foreach my $person (@people) {
>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>> print MAIL $mailSTR;
>>>> close(MAIL);
>>>> }
>>>>
>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>
>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>> $fileOBJ->sql....
>>>>
>>>> # create LOGGING Objects to log stuff
>>>> my $logOBJ = new ...
>>>> $logOBJ->sql...
>>>>
>>>> CORE::exit(0);
>>>> }
>>>>
>>>> print $in->redirect... # For the parent to redirect the browser
>>>>
>>>> # Done.
>>>>
>>>> Is there a glaring mistake in the above?
>>>>
>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>
>>>> This runs under ModPerl::Registry.
>>>>
>>>> I'd like to get at least one thing working tonight, either the
>>>> forking or the DBI, I'll be happy!
>>>>
>>>> Tosh
>>>>
>>>>
>>>>
>>>> Perrin Harkins wrote:
>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>> wrote:
>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and then
>>>>>> afterwards
>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>
>>>>>> [apache2]
>>>>>>
>>>>>> Is that what's supposed to happen?
>>>>> After you call exit? No. It should be gone. That's a zombie
>>>>> process.
>>>>>> That PM thread seems to indicate that I must disconnect every DBH,
>>>>>> not just
>>>>>> the ones that I will use.
>>>>> Either that, or you need to set InactiveDestroy on all of them in the
>>>>> child process. Otherwise, when the child exits, it messes up all of
>>>>> them for the parent.
>>>>>> Are you also suggesting the use of
>>>>>> Parallel::ForkManager for forks?
>>>>> No. The DBI stuff is the same with either.
>>>>> - Perrin
>>>>
>>>> --
>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: mod_perl2 + fork + DBI = Chaos

On Wed, Jan 27, 2010 at 8:22 AM, Tosh Cooey wrote:
> In case anyone else is ever in the same situation I would like to tell them
> that mod_perl(1|2) + fork = bad idea, and don't even THINK about throwing
> DBI into the mix.

I don't want people to think this doesn't work. It does work, and has
been used successfully before. Any time you mix DBI with forking, you
do have to be careful about closing handles, as described in the links
earlier in this thread. It's hard enough to get right that I would
advise people to use a job queue instead if possible.

> For me in the future, if I have a VERY long task I will externalize it, and
> if I have brief but long tasks then the following worked great for me:

That's good advice: use a separate job queue or a cleanup handler for
short tasks.

- Perrin

Re: 302 Redirect not working as expected with PerlCleanupHandler and Firefox under ModPerl::Registry

Just wanted to note that since you've put the CONN_CLOSE in the
redirect code, it's not necessary (nor desirable) to put
> "KeepAlive Off" in apache2.conf

With the CONN_CLOSE call you turn KA off just when you need it to be
off.

So what's the bad news?

cmac


On Jan 27, 2010, at 5:08 AM, Tosh Cooey wrote:

> The good news is that Mr. Mackenna got it! If I set "KeepAlive
> Off" in apache2.conf then it all works fine. Below is a
> functioning long process thingy which works with "KeepAlive On" and
> Firefox. I just hope it works with MSIE ...
>
>
> #!/usr/bin/perl
>
> use strict;
> use Apache2::Const -compile => qw(:conn_keepalive);
> use Apache2::Log(); # defines warn
> use Apache2::RequestUtil(); # defines push_handlers
> use Apache2::Connection();
> use Apache2::RequestRec();
>
> my $r = shift;
> my $c = $r->connection();
> $c->keepalive(Apache2::Const::CONN_CLOSE);
> $r->push_handlers(PerlCleanupHandler => \&cleanup );
>
> $r->err_headers_out->set(Location => 'http://...index.pl');
> $r->status(Apache2::Const::REDIRECT);
> return Apache2::Const::REDIRECT;
>
> sub cleanup {
> my ($r) = @_;
> $r->warn("Starting cleanup");
> foreach my $num (1..5) {
> $r->warn("Number is $num");
> sleep 2;
> }
> return Apache2::Const::OK;
> }
> ####
>
> So, thanks again, and as for the warning from William T., well my
> spin-off processes are maybe 10s-10m long and not a driving
> feature, but I will have to keep my eye on them.
>
> Tosh
>
>
> mackenna@animalhead.com wrote:
>> The warning from William T. made me think to ask:
>> Does your site have "KeepAlive On" in httpd.conf?
>> (If not I can't think of anything to suggest...)
>> If so, try adding this as part of the redirect:
>> use Apache2::Connection();
>> use Apache2::RequestRec();
>> ...
>> my $c = $r->connection();
>> $c->keepalive(Apache2::Const::CONN_CLOSE);
>> This will keep your process (which is about to do a "long"
>> cleanup) from automatically getting the redirected request
>> from the browser.
>> Hopefully the root httpd will know that this "redirecting"
>> child has not finished the complete cycle, and will launch
>> other children if needed to process the redirected request
>> plus any other requests.
>> Of course William is right that if lots of requests are
>> arriving that need such cleanup, and the cleanup really does
>> take a long time on average, you are likely to pile up
>> more children than your household income (I'm sorry I meant
>> to say your server :-) can support.
>> Good Luck,
>> cmac
>> On Jan 26, 2010, at 2:27 PM, Tosh Cooey wrote:
>>> So this works almost perfectly... Almost because:
>>>
>>> #!/usr/bin/perl
>>>
>>> use strict;
>>> use Apache2::Const(); # defines OK
>>> use Apache2::Log(); # defines warn
>>> use Apache2::RequestUtil(); # defines push_handlers
>>>
>>> my $r = shift;
>>> $r->push_handlers(PerlCleanupHandler => \&cleanup );
>>>
>>> $r->headers_out->set(Location => 'http://...index.pl');
>>> $r->status(Apache2::Const::REDIRECT);
>>> return Apache2::Const::REDIRECT;
>>>
>>> sub cleanup {
>>> my ($r) = @_;
>>> $r->warn("Starting cleanup");
>>> foreach my $num (1..5) {
>>> $r->warn("Number is $num");
>>> sleep 2;
>>> }
>>> return Apache2::Const::OK;
>>> }
>>> ######## test.pl
>>>
>>>
>>> It seems if you take the above program and hit it with Firefox
>>> (3.5.7 and 3.6) it may take 10 seconds (5 x sleep 2) before
>>> Firefox does the redirect. Safari 4.0.4 seems fine. curl works
>>> as well :)
>>>
>>> I said "may" above because it's not consistent. If you launch
>>> Firefox fresh and hit the above program it may redirect
>>> instantly, but then subsequent hits will illustrate the delay.
>>> I'm also seeing varying behaviour on a different server that has
>>> no Basic Auth, but always the problem is there.
>>>
>>> Can anyone else reproduce this?
>>>
>>> Thank-you!
>>>
>>> Tosh
>>>
>>>
>>> mackenna@animalhead.com wrote:
>>>> at(1) is a Unix command to start a process.
>>>> Assuming you're on a Unix/Linux box, type "man at" to get the
>>>> story.
>>>> A cleanup handler is more pleasant than a prostate exam.
>>>> You can spend your life waiting for others. Just write a
>>>> routine called "cleanup" and have it do something like
>>>> make a log entry.
>>>> use Apache2::Const(); # defines OK
>>>> use Apache2::Log(); # defines warn
>>>> use Apache2::RequestUtil(); # defines push_handlers
>>>> ...
>>>> sub cleanup {
>>>> my ($r) = @_;
>>>> $r->warn("cleanup was here");
>>>> return Apache2::Const::OK;
>>>> }
>>>> Then put a call like the one below in your ModPerl::Registry
>>>> routine.
>>>> If the log entry shows up in error_log, you're on your way...
>>>> Good Luck,
>>>> cmac
>>>> P.S. Google doesn't index some sites well.
>>>> Look at http://perl.apache.org/docs/2.0/
>>>> particularly its API link.
>>>> On Jan 25, 2010, at 5:49 PM, Tosh Cooey wrote:
>>>>> Sorry, I couldn't figure out what at(1) meant (or maybe ap(1)
>>>>> which you say below) is that an abbreviation for something?
>>>>>
>>>>> And Perrin saying "cleanup handler" is right up there with
>>>>> "prostate exam" in my list of things to get into, both scare me!
>>>>>
>>>>> Of course at some point a man needs to do both...
>>>>>
>>>>> So... If this magic: $r->push_handlers(PerlCleanupHandler =>
>>>>> \&cleanup); is available in ModPerl::Registry context then I
>>>>> will attempt to force all my forks into early retirement and
>>>>> work the problem out that way.
>>>>>
>>>>> Unfortunately Google doesn't return an easy answer, anybody
>>>>> know this before I spend all day tomorrow in my struggle?
>>>>>
>>>>> Thank-you all again,
>>>>>
>>>>> Tosh
>>>>>
>>>>>
>>>>> mackenna@animalhead.com wrote:
>>>>>> You made no comment on the links I sent you earlier today.
>>>>>> They had lots of good advice. Particularly the first one
>>>>>> suggested not forking the Apache process, but using an
>>>>>> ap(1) call to start a process to do the additional processing.
>>>>>> OK, the ap(1) alternative was a bit light on details.
>>>>>> How about the alternative offered by Perrin Hawkins in the
>>>>>> same thread, of using a cleanup handler to do the follow-up
>>>>>> processing rather than a forked process.
>>>>>> From p. 107 of "mod_per2 User's Guide":
>>>>>> $r->push_handlers(PerlCleanupHandler => \&cleanup);
>>>>>>> print $in->redirect... # to redirect the browser
>>>>>> Now cleanup (which receives $r as its operand) can do
>>>>>> whatever slow stuff you need to, can probably use DBI
>>>>>> without all the pain you have below, and can access the
>>>>>> request to find out what to do.
>>>>>> In some past context you may have learned how to get hold of
>>>>>> a $r to use in these calls, and hopefully you're no longer
>>>>>> scared of $r. But there does remain the question of whether
>>>>>> a ModPerl::Registry module can do such calls.
>>>>>> Hopefully someone who knows can chime in on this.
>>>>>> If not, for me it would be worth the editing of getting the
>>>>>> module out from under ModPerl::Registry and into the "native
>>>>>> mode" of SetHandler modperl.
>>>>>> Best of luck,
>>>>>> cmac
>>>>>> On Jan 25, 2010, at 1:54 PM, Tosh Cooey wrote:
>>>>>>> Ok, then maybe I need to supply some code here to try and get
>>>>>>> clarification:
>>>>>>>
>>>>>>> mailfile.pl
>>>>>>> ###########
>>>>>>> use strict;
>>>>>>> ...
>>>>>>> use POSIX;
>>>>>>>
>>>>>>> #gather needed modules and objects
>>>>>>> my $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>>>> my $clientOBJ = new ...
>>>>>>> my $userOBJ = new ...
>>>>>>> # All OBJjects have a {DBH} property which is their DB handle
>>>>>>> # I hear I have to disconnect these first, do I have to
>>>>>>> disconnect ALL?
>>>>>>> $fileOBJ->{DBH}->disconnect;
>>>>>>> $SIG{CHLD} = 'IGNORE';
>>>>>>> my $pid;
>>>>>>> if ($pid = fork) {
>>>>>>> warn "Pid = $pid";
>>>>>>> } elsif (defined $pid) {
>>>>>>> close(STDOUT);
>>>>>>> close(STDIN);
>>>>>>> close(STDERR);
>>>>>>>
>>>>>>> # chdir to /, stops the process from preventing an unmount
>>>>>>> chdir '/' or die "Can't chdir to /: $!";
>>>>>>> # dump our STDIN and STDOUT handles
>>>>>>> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
>>>>>>> open STDOUT, '>/dev/null' or die "Can't write to /dev/
>>>>>>> null: $!";
>>>>>>> # redirect for logging
>>>>>>> open STDERR, '>/tmp/stderr' or die "Can't write to /tmp/
>>>>>>> stderr: $!";
>>>>>>> # Prevent locking to apache process
>>>>>>> setsid or die "Can't start a new session: $!";
>>>>>>>
>>>>>>> # Create file download link email
>>>>>>> my $mailSTR = ...
>>>>>>>
>>>>>>> # Send the mail possibly to many people
>>>>>>> foreach my $person (@people) {
>>>>>>> open(MAIL, '|' . &cfg('sendmail_location') . ' -t');
>>>>>>> print MAIL $mailSTR;
>>>>>>> close(MAIL);
>>>>>>> }
>>>>>>>
>>>>>>> # Need to recreate the DBI connection on the $fileOBJ I hear
>>>>>>> $fileOBJ = new MyOBJS::FILE($in->param('id'));
>>>>>>>
>>>>>>> # Do some SQL to update the $fileOBJ status based on mailout
>>>>>>> $fileOBJ->sql....
>>>>>>>
>>>>>>> # create LOGGING Objects to log stuff
>>>>>>> my $logOBJ = new ...
>>>>>>> $logOBJ->sql...
>>>>>>>
>>>>>>> CORE::exit(0);
>>>>>>> }
>>>>>>>
>>>>>>> print $in->redirect... # For the parent to redirect the browser
>>>>>>>
>>>>>>> # Done.
>>>>>>>
>>>>>>> Is there a glaring mistake in the above?
>>>>>>>
>>>>>>> The parent does no more DB stuff, it just sends a redirect.
>>>>>>>
>>>>>>> This runs under ModPerl::Registry.
>>>>>>>
>>>>>>> I'd like to get at least one thing working tonight, either
>>>>>>> the forking or the DBI, I'll be happy!
>>>>>>>
>>>>>>> Tosh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Perrin Harkins wrote:
>>>>>>>> On Mon, Jan 25, 2010 at 3:48 PM, Tosh Cooey
>>>>>>>> wrote:
>>>>>>>>> Thanks Perrin, the forking, my child got a PID of 30033 and
>>>>>>>>> then afterwards
>>>>>>>>> when I checked the processes (ps) for 30033 I see:
>>>>>>>>>
>>>>>>>>> [apache2]
>>>>>>>>>
>>>>>>>>> Is that what's supposed to happen?
>>>>>>>> After you call exit? No. It should be gone. That's a
>>>>>>>> zombie process.
>>>>>>>>> That PM thread seems to indicate that I must disconnect
>>>>>>>>> every DBH, not just
>>>>>>>>> the ones that I will use.
>>>>>>>> Either that, or you need to set InactiveDestroy on all of
>>>>>>>> them in the
>>>>>>>> child process. Otherwise, when the child exits, it messes
>>>>>>>> up all of
>>>>>>>> them for the parent.
>>>>>>>>> Are you also suggesting the use of
>>>>>>>>> Parallel::ForkManager for forks?
>>>>>>>> No. The DBI stuff is the same with either.
>>>>>>>> - Perrin
>>>>>>>
>>>>>>> --
>>>>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>>>>>> 1200group.com/
>>>>>
>>>>> --
>>>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>>>> 1200group.com/
>>>
>>> --
>>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.
>>> 1200group.com/
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/