double mod_perl initialization

I was looking into why our server's restarts take so long, and I
finally remembered that Apache runs its initialization step twice on
startup (http://tinyurl.com/krr25). This means that my startup.pl is
loaded twice, along with any modules that it loads.

So I moved startup.pl to startup_real.pl and put this in startup.pl:

# Only run our real startup script the second time.
#
use Apache2::ServerUtil ();
if (Apache2::ServerUtil::restart_count() > 1) {
require 'startup_real.pl';
}

And it shaved a bunch of time off our restart.

As I understand it, the sole purpose of this double initialization is
to make sure that graceful restarts will work. However, since I was a
young mod_perl lad I've been taught never to rely on graceful
restarts, and always to stop/start.

If I don't ever plan to use graceful restarts, and I believe that
smaller restart times are an unqualified Good, is there any reason why
I shouldn't ALWAYS use a script like the above? And is there any way
to avoid PerlModule modules from being loaded twice?

Thanks
Jon

Re: double mod_perl initialization

> If I don't ever plan to use graceful restarts, and I believe that
> smaller restart times are an unqualified Good, is there any reason why
> I shouldn't ALWAYS use a script like the above? And is there any way
> to avoid PerlModule modules from being loaded twice?

I do something pretty similar. I do the double load on 'start', single
load on 'restart', and I don't load the app on 'stop' / 'graceful-stop'.

I've altered apachectl to set $ENV{APACHE_MODE} to whatever 'command'
was given, eg start / stop / restart etc.

Then my startup script looks like this:

------------------------------------------------------------ ----
my $apache_mode = $ENV{APACHE_MODE} || '';

die "'graceful-restart' is buggy. "
. "Use 'apachectl graceful-stop; apachectl start' instead."
if $apache_mode eq 'graceful-restart';

return 1
if $apache_mode
&& ( $apache_mode eq 'restart'
&& Apache2::ServerUtil::restart_count() == 1
|| $apache_mode !~ /start/ );

setup_my_application();
--------------------------------------------------------- -------

There's only one small gotcha that I've found, and that occurs in this
situation:

- apache isn't running
- you do : apachectl restart
- there is a compile time bug, and apache dies

The gotcha is that you don't get any STDERR or log output in this
situation. However, you're only ever likely to encounter this in
development, and you soon recognise the issue.

clint

Re: double mod_perl initialization

>
> There's only one small gotcha that I've found, and that occurs in this
> situation:
>
> - apache isn't running
> - you do : apachectl restart
> - there is a compile time bug, and apache dies
>
> The gotcha is that you don't get any STDERR or log output in this
> situation. However, you're only ever likely to encounter this in
> development, and you soon recognise the issue.

Hm. No compile errors would be bad. But I put an error in one of my
modules (that only gets loaded the second time) and started apache,
and got error log output. I wonder what we're doing differently?

Jon

Re: double mod_perl initialization

> Hm. No compile errors would be bad. But I put an error in one of my
> modules (that only gets loaded the second time) and started apache,
> and got error log output. I wonder what we're doing differently?

Shut down apache, then do "apachectl restart"


>
> Jon
>

Re: double mod_perl initialization

Nope, I still get the error in the logs. It just takes a few seconds
to show up.

On Sep 25, 2009, at 11:20 AM, Clinton Gormley wrote:

>
>> Hm. No compile errors would be bad. But I put an error in one of my
>> modules (that only gets loaded the second time) and started apache,
>> and got error log output. I wonder what we're doing differently?
>
> Shut down apache, then do "apachectl restart"
>
>
>>
>> Jon
>>
>

Re: double mod_perl initialization

On Sep 25, 2009, at 10:36 AM, Jonathan Swartz wrote:

> As I understand it, the sole purpose of this double initialization
> is to make sure that graceful restarts will work. However, since I
> was a young mod_perl lad I've been taught never to rely on graceful
> restarts, and always to stop/start.

This happens right along:
* a bug is found in a piece of software,
* the bug gets publicized and responded to and remembered by many
people,
* the people maintaining the software fix the bug,
* but for months and years into the future, people remember the bug
and keep working around it.

Apache's graceful restart worked fine years ago, but started losing
badly at some point.

I probably wasn't the only person to write a script called 'graceful'
to use instead of 'apachectl graceful'. My log rotation script still
uses a variant of 'graceful', but that's mostly because it determines
when it's OK to process the log.

But at some point, several iterative fixes by the Apache crew had
succeeded such that 'apachectl graceful' worked better than my
'graceful' script in some way.

It was sufficiently long ago that I've forgotten in what way, but the
memory is clear enough that I think Messrs. Swartz and Clint are
beating a dead bug.

I use 'apachectl graceful' a lot, and can't remember a problem for a
long time. Of course problems still exist with older versions of
apache2. If they work, graceful restarts are a good thing in that
they're as nice as possible to users on a production web site.

Double initialization does not prevent problems with restart
(graceful or otherwise) for most configuration or perl problems.
Isn't that because by the time the root httpd starts the first (re)
initialization it has already notified the children (that use the old
configuration and code) to go away? So if restart is the reason for
double initialization, by all means let's encourage the Apache folks
to reconsider the idea, and work around it until they do.

cmac

P.S. sudo's were omitted for brevity.

Re: double mod_perl initialization

> But at some point, several iterative fixes by the Apache crew had
> succeeded such that 'apachectl graceful' worked better than my
> 'graceful' script in some way.
>
> It was sufficiently long ago that I've forgotten in what way, but the
> memory is clear enough that I think Messrs. Swartz and Clint are
> beating a dead bug.
>

I'm using apache 2.2.9, mod_ssl 2.2.9 and mod_perl 2.04, and I still
have problems with graceful-restart.

It works under certain conditions, but (i think) combined with mod_ssl,
my httpd children hang around indefinitely, unless I set
GracefulShutdownTimeout.

For this reason I use: apachectl graceful-stop; apachectl start

> Double initialization does not prevent problems with restart
> (graceful or otherwise) for most configuration or perl problems.
> Isn't that because by the time the root httpd starts the first (re)
> initialization it has already notified the children (that use the old
> configuration and code) to go away? So if restart is the reason for
> double initialization, by all means let's encourage the Apache folks
> to reconsider the idea, and work around it until they do.

Because apache (normally) starts up as root, then drops privileges, it
tries to confirm that it can restart as not-root.

In my setup, I know that it can do this, and so I'd rather avoid that
double init. Also, I'd rather avoid setting up my whole application
just to shut down apache.

So, YMMV, but it works well for me.

clint

Re: double mod_perl initialization

On Sep 26, 2009, at 4:17 AM, Clinton Gormley wrote:

> I'm using apache 2.2.9, mod_ssl 2.2.9 and mod_perl 2.04, and I still
> have problems with graceful-restart.

You are 4 releases behind. Download 2.2.13 and I bet graceful will
work for you. (Bet's off if you have something systemically difficult
w/r/t ssl.)

Throw in a mod_perl 2.0.5 from SVN: I've never seen such performance
as the system I built with 'optimize -Os' last night.

> In my setup, I know that it can do this, and so I'd rather avoid that
> double init.

I would too. I was defending graceful restart, not double init.

cmac

Re: double mod_perl initialization

> You are 4 releases behind. Download 2.2.13 and I bet graceful will
> work for you. (Bet's off if you have something systemically difficult
> w/r/t ssl.)

It appears that you may be right:

Changes with Apache 2.2.12

*) prefork: Fix child process hang during graceful restart/stop in
configurations with multiple listening sockets. PR 42829. [Joe Orton,
Jeff Trawick]

Nice! Will try it out when I have a chance.

>
> Throw in a mod_perl 2.0.5 from SVN: I've never seen such performance
> as the system I built with 'optimize -Os' last night.

What does this do and how are you specifying it? I don't see any mention
of it in the docs for mod_perl or perl. Is that zero-s or oh-s?

ta

clint

Re: double mod_perl initialization

You can find changes about graceful restarts in many of the
version release notes in apache2. My memory is that 2.2.11
was OK for me.

Letter -Os means "optimize for size":

http://gcc.gnu.org/onlinedocs/gcc-2.95.3/gcc_2.html#SEC10

is the oldest gcc manual I could find, so the option has been
around for a long time.

On many compilers (including apparently gcc-x86) it also does
better on speed, unless you use -O3 which can inline your
memory size up. You want to get a compiler guy annoyed,
suggest that his compiler's "optimize for size" is faster
than its "opt for speed".

For building perl, you add '-Doptimize=-Os' to the Configure
command line.

For building Apache, you put '-Os' in CFLAGS at configure time
(possibly with other CFLAGS you may like).

mod_perl inherits it from perl or apache.
Other XS (C) modules inherit it from perl.
You can see either -O or -g (debugging) in most cc commands.

I tried -Os last week on my one CPAN module (ExtUtils::MakeMaker
suggests it). It took the size down between 10% and 15%. The
perl binary shrank in that range too. The impact on the httpd
process size is less, but there's lots of data in there.

So far there's also less in my error_log, but that's probably
coincidence :-)

Enjoy,
cmac


On Sep 26, 2009, at 10:36 AM, Clinton Gormley wrote:

>
>> You are 4 releases behind. Download 2.2.13 and I bet graceful will
>> work for you. (Bet's off if you have something systemically
>> difficult
>> w/r/t ssl.)
>
> It appears that you may be right:
>
> Changes with Apache 2.2.12
>
> *) prefork: Fix child process hang during graceful restart/stop in
> configurations with multiple listening sockets. PR 42829.
> [Joe Orton,
> Jeff Trawick]
>
> Nice! Will try it out when I have a chance.
>
>>
>> Throw in a mod_perl 2.0.5 from SVN: I've never seen such performance
>> as the system I built with 'optimize -Os' last night.
>
> What does this do and how are you specifying it? I don't see any
> mention
> of it in the docs for mod_perl or perl. Is that zero-s or oh-s?

Re: double mod_perl initialization

On Sep 25, 2009, at 10:48 AM, Clinton Gormley wrote:

>
>> If I don't ever plan to use graceful restarts, and I believe that
>> smaller restart times are an unqualified Good, is there any reason
>> why
>> I shouldn't ALWAYS use a script like the above? And is there any way
>> to avoid PerlModule modules from being loaded twice?
>
> I do something pretty similar. I do the double load on 'start', single
> load on 'restart', and I don't load the app on 'stop' / 'graceful-
> stop'.

Ok, one more question. Disregarding graceful for the moment - is HUP
completely reliable with mod_perl at this point, or is there still a
reason (as there once supposedly was) to do a full server stop and
start?

I've been doing stop and start reflexively for years. If it's
unnecessary, I wish I could have those seconds back. :)

Jon

Re: double mod_perl initialization

On Fri, Oct 2, 2009 at 5:54 PM, Jonathan Swartz wrote:
> Ok, one more question. Disregarding graceful for the moment - is HUP
> completely reliable with mod_perl at this point, or is there still a reason
> (as there once supposedly was) to do a full server stop and start?

The problem, at least in the 1.x series, was that the parent process
and the perl interpreter never restart. It just re-reads the config
file and runs through startup again.

- Perrin

Re: double mod_perl initialization

On Oct 2, 2009, at 3:25 PM, Perrin Harkins wrote:

> On Fri, Oct 2, 2009 at 5:54 PM, Jonathan Swartz
> wrote:
>> Ok, one more question. Disregarding graceful for the moment - is HUP
>> completely reliable with mod_perl at this point, or is there still
>> a reason
>> (as there once supposedly was) to do a full server stop and start?
>
> The problem, at least in the 1.x series, was that the parent process
> and the perl interpreter never restart. It just re-reads the config
> file and runs through startup again.

Right. Ok, in a *relatively modern Apache/mod_perl 2.x*, is there
still a reason to do a full server stop and start?

Thanks
Jon

Re: double mod_perl initialization

Been away from my computer for several days :(
Doesn't look like anyone is going to answer this.

Maybe you missed some of the emails that Clinton
Gormley and I were exchanging a while back. I've
been using 'sudo apachectl graceful' on apache
2.2.11 and 2.2.13 with mod_perl 2.0.4 and 2.0.5 SVN
for many months now and have not noticed any problems.

Of course you might be using some feature or module
or ??? that my site doesn't, which would cause some
problems. Fancy SSL use and changing certificates
were mentioned as possibilities.

If you look thru the change logs for the various apache
versions you can see quite a few improvements for
graceful and other restarts (there was one in 2.2.12).

So why not give them a chance, and if you have problems
come back here and tell us? Or look into them a little
and come back and tell us even more? And if graceful
restart works OK for you, please send an email to say
that.

The elimination of superstition is an absolute good.
cmac

On Oct 2, 2009, at 3:38 PM, Jonathan Swartz wrote:
>
> On Oct 2, 2009, at 3:25 PM, Perrin Harkins wrote:
>
>> On Fri, Oct 2, 2009 at 5:54 PM, Jonathan Swartz
>> wrote:
>>> Ok, one more question. Disregarding graceful for the moment - is HUP
>>> completely reliable with mod_perl at this point, or is there
>>> still a reason
>>> (as there once supposedly was) to do a full server stop and start?
>>
>> The problem, at least in the 1.x series, was that the parent process
>> and the perl interpreter never restart. It just re-reads the config
>> file and runs through startup again.
>
> Right. Ok, in a *relatively modern Apache/mod_perl 2.x*, is there
> still a reason to do a full server stop and start?
>
> Thanks
> Jon