ModPerl::RegistryLoader

Hello,

The documentation for "ModPerl::Registry" states: "Note that each httpd
process or "child" must compile each script once, so the first request
to one server may seem slow, but each request there after will be
faster. If your scripts are large and/or make use of many Perl modules,
this difference should be noticeable to the human eye."

I've indeed run into this as my service is not accessed so often so
typically the user requests reaches some httpd child which did not
compile the script yet. So user usually has to wait for the compilation.
In my case the difference is huge (2s versus tenths of ms) which makes
notable difference for the user experience for the various autocomplete
inputs etc.

So I was looking a solution and found ModPerl::RegistryLoader. I'm just
not sure I understand it completely. Is this the "compile once at
startup" solution? Do I just put " my $rlbb =
ModPerl::RegistryLoader->new();" into my startup file?


Thank you,
Jiri

RE: ModPerl::RegistryLoader

I don't know about ModPerl::RegistryLoader, but the simplest solution is ju=
st to "preload" the interpreters either manually or through a script whenev=
er you restart Apache. In your httpd.conf file, you control how many interp=
reters are running and how many hits each interpreter handles before reload=
ing that interpreter (it can be unlimited if you wish). If, e.g., you're co=
nfigured to run 6 interpreters, to be sure they're all loaded, you will nee=
d to send 6 concurrent requests to the server - if you send requests serial=
ly, it might use the same interpreter to serve all 6 requests. In our case,=
we've set up a web page that is a frames page with multiple frames, each m=
aking the same request to the server. We also have a script that utilizes L=
WP::UserAgent to do basically the same thing.=0A=
=0A=
FYI, our application consists of a single script which, depending on the pa=
rameters passed to it, can do many different things. Because of that, once =
that one script is loaded in an interpreter, there's nothing else to load. =
If you have many separate scripts, you can either do the above, but for eac=
h script. However, usually if you have many scripts, each script is quite s=
mall, but may load many Perl libraries (i.e. modules). Keep in mind that if=
you load a single script that loads, e.g. 10 Perl libraries, then when ano=
ther script is required that uses some of those libraries, Perl will detect=
the libraries that have already been loaded and not load them again. For t=
hat reason, the other scripts should load very quickly so it may be suffici=
ent to just preload the script that uses the most of these common libraries=
..=0A=
=0A=
An httpd.conf configuration we often use is something like:=0A=
=0A=
# Set up the settings for interpreters within mod_perl=0A=
=0A=
PerlInterpStart 12=0A=
PerlInterpMax 12=0A=
PerlInterpMinSpare 12=0A=
PerlInterpMaxSpare 12=0A=
PerlInterpMaxRequests 2048=0A=
=0A=
We've found that having a fixed number of interpreters at all times works b=
est for us - we've had problems in the past where an interpreter is retired=
by Apache when the load temporarily drops, only to be started up when the =
load picks back up (usually within a very short amount of time), requiring =
a full reload and slowing down response. That's why the first 4 settings ar=
e all the same. You might want to look it up, but I believe that if PerlInt=
erpMaxRequests is set to 0, an interpreter will take an unlimited number of=
hits without reloading. Reloading an interpreter, however, can help if you=
have memory leaks or other such "bad" things that accumulate over time.=0A=
________________________________________=0A=
From: Jiøí Pavlovsk=FD [jira@getnet.cz]=0A=
Sent: Thursday, July 21, 2011 4:45 AM=0A=
To: modperl@perl.apache.org=0A=
Subject: ModPerl::RegistryLoader=0A=
=0A=
Hello,=0A=
=0A=
The documentation for "ModPerl::Registry" states: "Note that each httpd=0A=
process or "child" must compile each script once, so the first request=0A=
to one server may seem slow, but each request there after will be=0A=
faster. If your scripts are large and/or make use of many Perl modules,=0A=
this difference should be noticeable to the human eye."=0A=
=0A=
I've indeed run into this as my service is not accessed so often so=0A=
typically the user requests reaches some httpd child which did not=0A=
compile the script yet. So user usually has to wait for the compilation.=0A=
In my case the difference is huge (2s versus tenths of ms) which makes=0A=
notable difference for the user experience for the various autocomplete=0A=
inputs etc.=0A=
=0A=
So I was looking a solution and found ModPerl::RegistryLoader. I'm just=0A=
not sure I understand it completely. Is this the "compile once at=0A=
startup" solution? Do I just put " my $rlbb = =
ModPerl::RegistryLoader->new();" into my startup file?=0A=
=0A=
=0A=
Thank you,=0A=
Jiri=0A=

Re: ModPerl::RegistryLoader

2011/7/21 Jiří Pavlovský :
> So I was looking a solution and found ModPerl::RegistryLoader. I'm just n=
ot
> sure I understand it completely. Is this the "compile once at startup"
> solution?

Yes, that's what it's for.

> Do I just put " my $rlbb =3D ModPerl::RegistryLoader->new();" into
> my startup file?

Well, you need more than that. Please read the man page and try it
out, and if you get stuck, come back and ask questions about the
specifics.

- Perrin

Re: ModPerl::RegistryLoader

On 21.7.2011 18:41, Perrin Harkins wrote:
>> So I was looking a solution and found ModPerl::RegistryLoader. I'm just not
>> sure I understand it completely. Is this the "compile once at startup"
>> solution?
> Yes, that's what it's for.
Thanks, so I'm on correct track
>
>> Do I just put " my $rlbb = ModPerl::RegistryLoader->new();" into
>> my startup file?
> Well, you need more than that. Please read the man page and try it
> out, and if you get stuck, come back and ask questions about the
> specifics.

I did RTFM, but the man page was not clear to me, so I asked on the list.

Anyway, now I tried to implement something along the lines suggested in
the synopsis:

sub trans {
my $uri = shift;
return File::Spec->catfile(MY_DOC_ROOT, $uri);
}
my $rl = ModPerl::RegistryLoader->new(
package => 'ModPerl::RegistryPrefork',
trans => \&trans,
);

.....
# these $apps are of the form '/cgi-bin/script.pl'
foreach my $app (@apps) {
$rl->handler($app);
}

In addition I've few Location directives in httpd.conf to load few other
scripts.

However this doesn't work as Apache is segfaulting.

Re: ModPerl::RegistryLoader

2011/7/21 Jiří Pavlovský :
> I did RTFM, but the man page was not clear to me, so I asked on the list.

Asking on the list is encouraged! We just need specific questions

> However this doesn't work as Apache is segfaulting.

That makes it sound like it is working but there's something in your
scripts that doesn't like being loaded in the parent process. Are you
opening up database handles or files on the first hit and then trying
to use them later?

- Perrin

Re: ModPerl::RegistryLoader

Hi Jiri,

i am not sure what you are trying to do with ModPerl::RegistryLoader,
but what me helped to improve performance was :

1. Preload almost every library on startup:
PerlPostConfigRequire /xyz/startup.pl (

---------
use Apache2::RequestRec();
use Apache2::RequestIO();
use Apache2::RequestUtil();
use Apache2::ServerRec();
use Apache2::ServerUtil();
use Apache2::Connection();
use Apache2::Log();
use Apache2::Request();
use APR::Table();
#... and so on
----------
2. reused db connection over requests

3. using mmap cache for Data that comes from db

4. last but not least use HTTP cache Protocol with mod_cache

All together hundred of projects with hundreds of simultaneous requests
are possible on only one server.
regards Keywan

Am 21.07.2011 21:58, schrieb Jiří Pavlovský:
> On 21.7.2011 18:41, Perrin Harkins wrote:
>>> So I was looking a solution and found ModPerl::RegistryLoader. I'm
>>> just not
>>> sure I understand it completely. Is this the "compile once at startup"
>>> solution?
>> Yes, that's what it's for.
> Thanks, so I'm on correct track
>>
>>> Do I just put " my $rlbb = ModPerl::RegistryLoader->new();" into
>>> my startup file?
>> Well, you need more than that. Please read the man page and try it
>> out, and if you get stuck, come back and ask questions about the
>> specifics.
>
> I did RTFM, but the man page was not clear to me, so I asked on the list.
>
> Anyway, now I tried to implement something along the lines suggested
> in the synopsis:
>
> sub trans {
> my $uri = shift;
> return File::Spec->catfile(MY_DOC_ROOT, $uri);
> }
> my $rl = ModPerl::RegistryLoader->new(
> package => 'ModPerl::RegistryPrefork',
> trans => \&trans,
> );
>
> ....
> # these $apps are of the form '/cgi-bin/script.pl'
> foreach my $app (@apps) {
> $rl->handler($app);
> }
>
> In addition I've few Location directives in httpd.conf to load few
> other scripts.
>
> However this doesn't work as Apache is segfaulting.
>
>
>


--
*Keywan Ghadami*
Entwicklung und Sicherheit

*ibson*
Fritz-Arnold-Str. 23
D-78467 Konstanz

Telefon +49 (0)75 31 / 81 39 7 93
Fax +49 (0)75 31 / 81 39 7 89

Re: ModPerl::RegistryLoader

On 22.7.2011 18:15, Perrin Harkins wrote:
>> However this doesn't work as Apache is segfaulting.
> That makes it sound like it is working but there's something in your
> scripts that doesn't like being loaded in the parent process. Are you
> opening up database handles or files on the first hit and then trying
> to use them later?
>

I use Apache::DBI and initialize the db connection in startup.pl like so

Apache::DBI->connect_on_init("dbi:Pg:dbname=",
"",
"",
{
pg_server_prepare => 1,
PrintError => 0,
RaiseError => 0,
AutoCommit => 1,
pg_enable_utf8 => 1,
Taint => 0,
}
);

Re: ModPerl::RegistryLoader

2011/7/22 Jiří Pavlovský :
> On 22.7.2011 18:15, Perrin Harkins wrote:
>>>
>>> However this doesn't work as Apache is segfaulting.
>>
>> That makes it sound like it is working but there's something in your
>> scripts that doesn't like being loaded in the parent process.  Are =
you
>> opening up database handles or files on the first hit and then trying
>> to use them later?
>>
>
> I use Apache::DBI and initialize the db connection in startup.pl like so

That's good, but you might be storing a database handle in a
persistent global of closure variable somewhere, and that could lead
to a segfault.

It's hard for me to give you detailed debugging help on this, but my
basic advice is that if you have repeatable test (i.e. it segfaults
every time), you just need to comment out code until it stops
segfaulting and you'll find the culprit.

If you want to make sure it's your code and not RegistryLoader itself,
try just preloading an empty script that does nothing. It that
doesn't segfault, then the problem is in your code and it's
interaction with being loaded in the parent process.

- Perrin

Re: ModPerl::RegistryLoader

I am not sure if this directly applies to your issue, but I got segfaults
in DBI when I just opened the DB handle once in the parent, reusing the
DBH in child processes (a bug). The issue went away when I ensured that
every child process did its own DBI->connect.
HTH


On 7/22/11 11:17 AM, "Jiøí Pavlovsk=FD" wrote:

>I use Apache::DBI and initialize the db connection in startup.pl like so
>

Re: ModPerl::RegistryLoader

On Monday, 25 July 2011 23:31:04 McCarrell, Jeff wrote:
> I am not sure if this directly applies to your issue, but I got
> segfaults in DBI when I just opened the DB handle once in the parent,
> reusing the DBH in child processes (a bug). The issue went away when
> I ensured that every child process did its own DBI->connect.

That is what connect_on_init tries to achieve.

Are you by chance using a threaded MPM? If yes, it may be the reason.=20
connect_on_init works as a PerlChildInitHandler. This phase runs once for=20
each Apache child process and not once per Perl interpreter.

As a remedy it shouldn't hurt much to comment this line out except for=20
the first request that uses the database. Postgres normally connects=20
quite fast. So I think it wouldn't be really noticable.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: ModPerl::RegistryLoader

On 25.7.2011 22:18, Perrin Harkins wrote:
> 2011/7/22 Jiří Pavlovský:
>> On 22.7.2011 18:15, Perrin Harkins wrote:
>>>> However this doesn't work as Apache is segfaulting.
>>> That makes it sound like it is working but there's something in your
>>> scripts that doesn't like being loaded in the parent process. Are you
>>> opening up database handles or files on the first hit and then trying
>>> to use them later?
>>>
>> I use Apache::DBI and initialize the db connection in startup.pl like so
> That's good, but you might be storing a database handle in a
> persistent global of closure variable somewhere, and that could lead
> to a segfault.
That is probably it, I store db handle in a singleton.
That, unfortunately is something I cannot change easily

Re: ModPerl::RegistryLoader

2011/7/26 Jiří Pavlovský :
> That is probably it, I store db handle in a singleton.
> That, unfortunately is something I cannot change easily

Apache::DBI should make this unnecessary. You could change your
singleton code to DBI->connect every time and it would just pull the
cached handle from Apache::DBI.

Alternatively, you can try adjusting your code so that it just avoids
doing the DBI caching during startup. Look at the Apache::DBI code
for an example of how to tell if you're in startup.

- Perrin

Re: ModPerl::RegistryLoader

On 26.7.2011 16:08, Perrin Harkins wrote:
> 2011/7/26 Jiří Pavlovský:
>> That is probably it, I store db handle in a singleton.
>> That, unfortunately is something I cannot change easily
> Apache::DBI should make this unnecessary. You could change your
> singleton code to DBI->connect every time and it would just pull the
> cached handle from Apache::DBI.

Thanks. I'll try this.