Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

I want to build a mod_perl2 application using Sleepycat::DbXml. This is
the Perl bindings to the C++ interface to Berkeley DB and Berkeley DB
XML, developed by Sleepycat, now owned by Oracle (keywords: DbEnv,
XmlManager, XmlContainer).

Has anyone ever used this combination?

Note I'm using Apache 2.2 with the worker MPM and a suitably configured
Perl (perl -V:useithreads -V:usemultiplicity).

I want to open the environment and the XML container (the database)
during initialization so this doesn't have to be done anew on each
request.

However, I don't know how to do this. Currently, I'm trying to set up
things in startup.pl (loaded via PerlPostConfigRequire), store the
database environment handle in a global package variable, and access
that from the PerlContentHandler.

(1) Is that okay in principle?

(2) As I'm getting unfriendly SEGVs: Has anyone any experience on using
mod_perl2 with Sleepycat::Db or Sleepycat::DbXml? If so, I can provide
more details on this.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects instartup.pl

On Wed, 14 Jan 2009, Michael Ludwig wrote:

> I want to build a mod_perl2 application using
> Sleepycat::DbXml. This is
>
> However, I don't know how to do this. Currently, I'm
> trying to set up things in startup.pl (loaded via
> PerlPostConfigRequire), store the database environment
> handle in a global package variable, and access that from
> the PerlContentHandler.

This probably won't work since the filehandle cannot be
shared among mod_perl children.

Your startup.pl script is running before the root apache
process forks into the child processes. Scalars, lists and
hashes will be cloned, but file handles won't, they have to
be opened.

Probably what you're thinking of is a PerlChildInitHandler
so that each mod_perl child process does your connection for
you when the child process first forks.

Or, you can use it like this in a handler, essentially the
same thing:

package YourResponseHandler;
use strict;
use warnings FATAL => 'all';

use SleepyCat::DbXml qw(simple);

# is this the way it works? i trolled for it...
# where is the manual page?? ugh
my $container = XmlContainer->new('/path/to/whatever.xml');

sub handler {
my ($r) = @_;
$container->open(Db::DB_CREATE);
$container->put...
# ...
$container->close(); # ?? not sure how it works
}

But it's not clear how much time you're going to save. I
don't know how the thing works. You may or may not be able
to call open() from package scope because of locking, are
locks on the sleepycat database serial? If so then every
mod_perl child process is going to be waiting for the others
to complete before they can establish their own locks and
there's no way around that. You'll have to balance how many
writes you expect to get with how many reads, maybe reading
actions do not require locks on the data file.

Mark

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

This is my first time replying to the list. I've seen
advice about not being able to share a filehandle opened
in a pre-fork stage before, but have two counter-examples:

1) I opened a log file for write/append in the open-logs
stage of a module, and was able to (flock and) write to it
in child processes. (It's called the open-logs stage!)

2) My web site ties its read-only DBs at the post-config
stage and reads them in the child httpd's.

My one RW DB (for guestbook filtering) has an accompanying
lockfile that's opened and flocked before the DB is tied,
on the fly.

I have a vague recollection of reading about the
circumstances in which filehandles can be inherited, but
can't remember where. Programming Perl?

Sorry to make things more complex, but I'll be happy if
someone can clarify this.

cmac
www.animalhead.com

On Jan 14, 2009, at 8:18 AM, Mark Hedges wrote:
> On Wed, 14 Jan 2009, Michael Ludwig wrote:
>
>> I want to build a mod_perl2 application using
>> Sleepycat::DbXml. This is
>>
>> However, I don't know how to do this. Currently, I'm
>> trying to set up things in startup.pl (loaded via
>> PerlPostConfigRequire), store the database environment
>> handle in a global package variable, and access that from
>> the PerlContentHandler.
>
> This probably won't work since the filehandle cannot be
> shared among mod_perl children.
>
> Your startup.pl script is running before the root apache
> process forks into the child processes. Scalars, lists and
> hashes will be cloned, but file handles won't, they have to
> be opened.
>
> Probably what you're thinking of is a PerlChildInitHandler
> so that each mod_perl child process does your connection for
> you when the child process first forks.
>

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

craig@animalhead.com wrote:

> I have a vague recollection of reading about the
> circumstances in which filehandles can be inherited, but
> can't remember where.

I've been bitten by this a few times. Filehandles (and thus sockets) are inherited across forks. If
your system isn't very busy you won't notice a problem. But when things heat up you will start to
get very strange problems. And I do mean strange.

I was doing something with Inline::Java (which communicates over a socket to a JVM) under mod_perl
and the communication was getting all garbled. Multiple processes were sending and reading off the
socket which confused the other processes since they were all getting partial reads on their data
and partial reads from the data for other processes. But the problems only happened during the
busiest 2 weeks of the year and were fine the rest of the time. But those weren't very fun weeks :)

--
Michael Peters
Plus Three, LP

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Mark Hedges schrieb:
> Probably what you're thinking of is a PerlChildInitHandler
> so that each mod_perl child process does your connection for
> you when the child process first forks.

Yes, that's what I thought. But then I noticed that the
PerlChildInitHandler I set up is ignored. Apache configuration
snippet:

PerlRequire /home/milu/www/eumel/startup.pl
PerlChildInitHandler Eumel::Gurke::bla;
PerlChildExitHandler Eumel::Gurke2::bla;

These two classes do not even exist, but Apache starts fine.
When I replace those dummy entries with existing classes, I can
see they're ignored because I don't see the log output I should
see.

Is is possible these are ignored for the worker MPM?

| In the prefork MPM this phase is useful for initializing
| any data structures which should be private to each process.

http://perl.apache.org/docs/2.0/user/handlers/server.html#C_ PerlChildInitHandler_

It is unclear to me whether this applies to the prefork MPM
exclusively. Does anyone know?

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Mark Hedges schrieb:
> On Wed, 14 Jan 2009, Michael Ludwig wrote:
>
>> I want to build a mod_perl2 application using
>> Sleepycat::DbXml. This is
>>
>> However, I don't know how to do this. Currently, I'm
>> trying to set up things in startup.pl (loaded via
>> PerlPostConfigRequire), store the database environment
>> handle in a global package variable, and access that from
>> the PerlContentHandler.
>
> This probably won't work since the filehandle cannot be
> shared among mod_perl children.

Thanks for your help, Mark. I'm not sure it's a plain filehandle,
but probably your remark applies anyway.

> Your startup.pl script is running before the root apache
> process forks into the child processes. Scalars, lists and
> hashes will be cloned, but file handles won't, they have to
> be opened.

Cloning might not be what the DbEnv, XmlManager and XmlContainer
objects need, or tolerate.

> Or, you can use it like this in a handler, essentially the
> same thing [as the PerlChildInitHandler]:
>
> package YourResponseHandler;
> use strict;
> use warnings FATAL => 'all';
>
> use SleepyCat::DbXml qw(simple);
>
> # is this the way it works? i trolled for it...
> # where is the manual page?? ugh

Probably not all in place yet.

> my $container = XmlContainer->new('/path/to/whatever.xml');
>
> sub handler {
> my ($r) = @_;
> $container->open(Db::DB_CREATE);
> $container->put...
> # ...
> $container->close(); # ?? not sure how it works
> }

I tried this style of initialization. It doesn't solve things.
Depending on how exactly I code this, I get either a SEGV on
startup attempt or when attempting concurrent access to the
application. I also saw this:

*** glibc detected *** free(): invalid pointer: 0x084e6a14 ***

> But it's not clear how much time you're going to save. I
> don't know how the thing works. You may or may not be able
> to call open() from package scope because of locking, are
> locks on the sleepycat database serial?

No, it supports multi-process and multi-threaded access.

Good news: I've been offered help. I'll report back once I'll
have attained a successful configuration, or be convinced that
this simply won't work.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Apache is forgiving/robust about specifying nonexistent phase
handlers in httpd.conf and inserts thereto.

I was using event, and had to let each thread open its own DBs
based on an undefined global. Then I found that the identical
traffic-test exerciser could get 66 files/second back in event
and 1500/second in prefork. This is under FreeBSD 6.3.

No doubt I had something wrong or "tunable" in event, but at
that point I gave up and will wait for the simple MPM.

Best Regards,
cmac
www.animalhead.com


On Jan 15, 2009, at 7:49 AM, Michael Ludwig wrote:

> Mark Hedges schrieb:
>> Probably what you're thinking of is a PerlChildInitHandler
>> so that each mod_perl child process does your connection for
>> you when the child process first forks.
>
> Yes, that's what I thought. But then I noticed that the
> PerlChildInitHandler I set up is ignored. Apache configuration
> snippet:
>
> PerlRequire /home/milu/www/eumel/startup.pl
> PerlChildInitHandler Eumel::Gurke::bla;
> PerlChildExitHandler Eumel::Gurke2::bla;
>
> These two classes do not even exist, but Apache starts fine.
> When I replace those dummy entries with existing classes, I can
> see they're ignored because I don't see the log output I should
> see.
>
> Is is possible these are ignored for the worker MPM?
>
> | In the prefork MPM this phase is useful for initializing
> | any data structures which should be private to each process.
>
> http://perl.apache.org/docs/2.0/user/handlers/
> server.html#C_PerlChildInitHandler_
>
> It is unclear to me whether this applies to the prefork MPM
> exclusively. Does anyone know?
>
> Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects instartup.pl

On Thu, 15 Jan 2009, Michael Ludwig wrote:

> PerlRequire /home/milu/www/eumel/startup.pl
> PerlChildInitHandler Eumel::Gurke::bla;
> PerlChildExitHandler Eumel::Gurke2::bla;

No trailing semicolons?

Probably if you first do PerlLoadModule Foo::Bar it will
fail and inform you

> > my $container = XmlContainer->new('/path/to/whatever.xml');
> >
> > sub handler {
> > my ($r) = @_;
> > $container->open(Db::DB_CREATE);
> > $container->put...
> > # ...
> > $container->close(); # ?? not sure how it works
> > }
>
> I tried this style of initialization. It doesn't solve
> things. Depending on how exactly I code this, I get either
> a SEGV on startup attempt or when attempting concurrent
> access to the application. I also saw this:
>
> *** glibc detected *** free(): invalid pointer: 0x084e6a14 ***

If this didn't work in a response handler, I'd guess there's
something else wrong. That's the kind of error you get when
you use things under threads that are not thread-safe in
some way, I remember that sort of thing trying to use
List::MoreUtils::natatime() under threads. (But the rest of
that library works.) It's a sad state of affairs that a
large chunk of CPAN doesn't play nice with threads, and
there's usually not any way of knowing which library is
causing the problem. It might not even be the oracle
thingy, maybe something else you're loading.

Mark

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Mark Hedges schrieb:
>
> On Thu, 15 Jan 2009, Michael Ludwig wrote:
>
>> PerlRequire /home/milu/www/eumel/startup.pl
>> PerlChildInitHandler Eumel::Gurke::bla;
>> PerlChildExitHandler Eumel::Gurke2::bla;
>
> No trailing semicolons?

That was a copy and paste error when composing the mail. But this is an
important point, as there won't be any warning about this - it will
silently fail, regardless of whether or not the module has been loaded.

> Probably if you first do PerlLoadModule Foo::Bar it will
> fail and inform you

It definitely will fail and output an error message. So it's crucial
to PerlLoadModule the module first, else even a correctly specified
PerlChildExitHandler won't be installed.

Thanks a lot, Mark. More on this later.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

mackenna@animalhead.com schrieb:
> Apache is forgiving/robust about specifying nonexistent phase
> handlers in httpd.conf and inserts thereto.

Thanks. I noticed. :-)

> I was using event, and had to let each thread open its own DBs
> based on an undefined global. Then I found that the identical
> traffic-test exerciser could get 66 files/second back in event
> and 1500/second in prefork. This is under FreeBSD 6.3.
>
> No doubt I had something wrong or "tunable" in event, but at
> that point I gave up and will wait for the simple MPM.

Thanks. I'll try prefork if I can't get worker to work.

Michael

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

mackenna@animalhead.com wrote:
> Apache is forgiving/robust about specifying nonexistent phase
> handlers in httpd.conf and inserts thereto.
>
> I was using event, and had to let each thread open its own DBs
> based on an undefined global. Then I found that the identical
> traffic-test exerciser could get 66 files/second back in event
> and 1500/second in prefork. This is under FreeBSD 6.3.
>
> No doubt I had something wrong or "tunable" in event, but at
> that point I gave up and will wait for the simple MPM.

The event MPM is also currently referred to as being 'experimental' by
the httpd folks. have you tried worker instead?

Adam

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

I started with 'worker', and it had similar performance to
what I saw thereafter with 'event'. Actually slightly better,
perhaps because 'event' gained no advantage from KeepAlive
because each test process stayed locked to one connection.

'worker' and 'event' were experiments that I decided to
terminate, in part because I was so annoyed with Perl ithreads
after I had laboriously eliminated many global variables from
my mod_perl scripts that didn't want to be shared. This
activity was based on the "Programming Perl" description of
"threads 5.005". Oh well, now the code should be better set
for the future.

Best Regards,
cmac
www.animalhead.com


On Jan 16, 2009, at 8:14 AM, Adam Prime wrote:

> mackenna@animalhead.com wrote:
>> Apache is forgiving/robust about specifying nonexistent phase
>> handlers in httpd.conf and inserts thereto.
>> I was using event, and had to let each thread open its own DBs
>> based on an undefined global. Then I found that the identical
>> traffic-test exerciser could get 66 files/second back in event
>> and 1500/second in prefork. This is under FreeBSD 6.3.
>> No doubt I had something wrong or "tunable" in event, but at
>> that point I gave up and will wait for the simple MPM.
>
> The event MPM is also currently referred to as being 'experimental'
> by the httpd folks. have you tried worker instead?
>
> Adam

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Mark Hedges schrieb:
>>
>> *** glibc detected *** free(): invalid pointer: 0x084e6a14 ***
>
> If this didn't work in a response handler, I'd guess there's
> something else wrong. That's the kind of error you get when
> you use things under threads that are not thread-safe in
> some way, I remember that sort of thing trying to use
> List::MoreUtils::natatime() under threads. (But the rest of
> that library works.) It's a sad state of affairs that a
> large chunk of CPAN doesn't play nice with threads, and
> there's usually not any way of knowing which library is
> causing the problem. It might not even be the oracle
> thingy, maybe something else you're loading.

It may have been XML::LibXML, XML::LibXSLT, or - most probably - my
lack of understanding of the worker MPM and threaded Perl.

I made some progress and got it working using package-level
my-variables to cache the Berkeley handles.

Still, concurrent access quite reliably manages to produce SEGVs.
This may have something to do with with not finalizing the handles
properly.

So I think I have to use PerlChildInitHandler and PerlChildExitHandler.

I spent some time wondering why the variables I initialized in the
PerlChildInitHandler Eumel::A::child_init were undefined when
accessed from the PerlResponseHandler Eumel::B::handler, regardless
of whether they were declared with our or with my and then returned
by a function.

I thought that moving the child_init method into the same package
as the handler and having it initialize package level my-variables
or our-variables might do the trick. But it doesn't.

Looks like the variables that the PerlChildInitHandler sees are
different beasts than those the PerlResponseHandler sees, even
though they are textually identical.

Turns out under the threaded worker MPM things behave differently:

| Global variables are only global to the interpreter in which
| they are created. Other interpreters from other threads
| can't access that variable. Though it's possible to make
| existing variables shared between several threads running in
| the same process by using the function
| threads::shared::share().

http://perl.apache.org/docs/2.0/user/coding/coding.html#Shar ed_Variables

Should have RTFM before.

I haven't used threads to this date, nor threads::shared. Looks like
I'd have to share the Berkeley handles in order for it to *maybe*
work.

Now here's a funny one.

use threads;
use threads::shared;

our $env;

sub child_init {
share($env);

$env = DbEnv->new;
$env->open( $envdir, Db::DB_USE_ENVIRON | Db::DB_THREAD);
print STDERR "child_init: Umgebung geoeffnet\n";
...
}

I never see the log message - the PerlChildInitHandler thing is
hanging. Which bizarrely doesn't stop the PerlResponseHandler, by
the way.

If I uncomment "use threads", the threads::shared stuff conveniently
turns into no-ops, and I get back the old behaviour, which is that
the PerlChildInitHandler and the PerlResponseHandler do not see the
same variables in spite of their being textually identical.

I'm at my wit's end. Any clues?

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

What I did to get worker and event working under FreeBSD 6.3,
was to eliminate the child_init handler, and at the start of
the response handler do something like

> my $env;

sub handler { # this is the response handler
my ($r) = @_;
if (!$env) {
>> $env = DbEnv->new;
>> $env->open( $envdir, Db::DB_USE_ENVIRON | Db::DB_THREAD);
...
> }
...
}

What this means is that each thread must open the db's for itself.
The amount of data stored for each open DB connection, times
THREADS_PER_CHILD times the number of Apache children at any
given point, makes for some memory. But

1) the separate connections help the DB package be thread-safe,
2) the first-used threads keep getting re-used in preference to
threads not yet used.
3) if you consider each thread as more or less equivalent to a
child process in prefork, your total memory requirement is less.

Guten abend (in meiner Zeitzone),
cmac
www.animalhead.com


On Jan 16, 2009, at 11:34 AM, Michael Ludwig wrote:

> Mark Hedges schrieb:
>>>
>>> *** glibc detected *** free(): invalid pointer: 0x084e6a14 ***
>>
>> If this didn't work in a response handler, I'd guess there's
>> something else wrong. That's the kind of error you get when
>> you use things under threads that are not thread-safe in
>> some way, I remember that sort of thing trying to use
>> List::MoreUtils::natatime() under threads. (But the rest of
>> that library works.) It's a sad state of affairs that a
>> large chunk of CPAN doesn't play nice with threads, and
>> there's usually not any way of knowing which library is
>> causing the problem. It might not even be the oracle
>> thingy, maybe something else you're loading.
>
> It may have been XML::LibXML, XML::LibXSLT, or - most probably - my
> lack of understanding of the worker MPM and threaded Perl.
>
> I made some progress and got it working using package-level
> my-variables to cache the Berkeley handles.
>
> Still, concurrent access quite reliably manages to produce SEGVs.
> This may have something to do with with not finalizing the handles
> properly.
>
> So I think I have to use PerlChildInitHandler and
> PerlChildExitHandler.
>
> I spent some time wondering why the variables I initialized in the
> PerlChildInitHandler Eumel::A::child_init were undefined when
> accessed from the PerlResponseHandler Eumel::B::handler, regardless
> of whether they were declared with our or with my and then returned
> by a function.
>
> I thought that moving the child_init method into the same package
> as the handler and having it initialize package level my-variables
> or our-variables might do the trick. But it doesn't.
>
> Looks like the variables that the PerlChildInitHandler sees are
> different beasts than those the PerlResponseHandler sees, even
> though they are textually identical.
>
> Turns out under the threaded worker MPM things behave differently:
>
> | Global variables are only global to the interpreter in which
> | they are created. Other interpreters from other threads
> | can't access that variable. Though it's possible to make
> | existing variables shared between several threads running in
> | the same process by using the function
> | threads::shared::share().
>
> http://perl.apache.org/docs/2.0/user/coding/
> coding.html#Shared_Variables
>
> Should have RTFM before.
>
> I haven't used threads to this date, nor threads::shared. Looks like
> I'd have to share the Berkeley handles in order for it to *maybe*
> work.
>
> Now here's a funny one.
>
> use threads;
> use threads::shared;
>
> our $env;
>
> sub child_init {
> share($env);
> $env = DbEnv->new;
> $env->open( $envdir, Db::DB_USE_ENVIRON | Db::DB_THREAD);
> print STDERR "child_init: Umgebung geoeffnet\n";
> ...
> }
>
> I never see the log message - the PerlChildInitHandler thing is
> hanging. Which bizarrely doesn't stop the PerlResponseHandler, by
> the way.
>
> If I uncomment "use threads", the threads::shared stuff conveniently
> turns into no-ops, and I get back the old behaviour, which is that
> the PerlChildInitHandler and the PerlResponseHandler do not see the
> same variables in spite of their being textually identical.
>
> I'm at my wit's end. Any clues?
>
> Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

mackenna@animalhead.com schrieb:
> What I did to get worker and event working under FreeBSD 6.3,
> was to eliminate the child_init handler, and at the start of
> the response handler do something like
>
> my $env;
>
> sub handler { # this is the response handler
> my ($r) = @_;
> if (!$env) {
> $env = DbEnv->new;
> $env->open( $envdir, Db::DB_USE_ENVIRON | Db::DB_THREAD);
> ...
> }
> ...
> }

Guten Abend, Craig!

Thanks for replying again. What you've outlined above is what I had
implemented following Mark's suggestion in his first reply to my initial
post. Unfortunately, this quite reliably produces SEGVs, so I think this
usage is not aligned with the Berkeley interface design. I don't know in
which way this is wrong - I guess I'll have to ask the Berkeley folks in
order to maybe clarify this.

> What this means is that each thread must open the db's for itself.
> The amount of data stored for each open DB connection, times
> THREADS_PER_CHILD times the number of Apache children at any
> given point, makes for some memory. But
>
> 1) the separate connections help the DB package be thread-safe,

So if coding the handler as above means that each thread, having its own
global variables, opens its own handle to the Berkeley environment, I
shouldn't need the DB_THREAD flag (which, according to the Berkeley
documentation, "[causes] the DbEnv handle returned by DbEnv::open to be
free-threaded; that is, concurrently usable by multiple threads in the
address space." This flag is needed if any handle is used by more than
one thread (or process) concurrently. So it shouldn't be needed if the
handler is coded as above and there isn't any concurrent access
elsewhere.

> 2) the first-used threads keep getting re-used in preference to
> threads not yet used.

I noticed this is indeed what seems to happen - whether by chance or as
a feature, I don't know.

> 3) if you consider each thread as more or less equivalent to a
> child process in prefork, your total memory requirement is less.

From perldoc perlthrtut: "In this model each thread runs in its own Perl
interpreter, and any data sharing between threads must be explicit."
This does not sound to me as if there is a significant advantage over
spawning child processes, at least not on UNIX.

Hmm. Not sure what to make of this threaded Perl.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

I've referenced this thread on the Oracle Berkeley DB forum.
Chances are it'll catch the attention of one of the BDB experts.
If you're interested, please see:

http://forums.oracle.com/forums/thread.jspa?threadID=848390

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

On Sun 18 Jan 2009, Michael Ludwig wrote:
> =A0From perldoc perlthrtut: "In this model each thread runs in its own
> Perl interpreter, and any data sharing between threads must be
> explicit." This does not sound to me as if there is a significant
> advantage over spawning child processes, at least not on UNIX.
>
> Hmm. Not sure what to make of this threaded Perl.

In fact, it is worse than fork()ing off unix processes because the=20
interpreter data is completely copied while a new interpreter is=20
created. On the other hand a forked process copies only those pages=20
that are written to.

I haven't used Sleepycat::DbXml but I use BerkeleyDB in production with=20
prefork, see Apache2::Translation on CPAN. What I have learned is, you=20
can't open the BDB env in the parent apache. When I wrote that piece I=20
also had problems when I opened the env in the parent and then closed=20
it before forking children. Perhaps they are gone in a newer version.

I think with prefork opening the env in a ChildInit handler should work.=20
I ended up with the connect-at-first-usage approach

unless( $connected ) {
connect();
$connected=3D1;
}

because I test almost all my modules also with a threaded MPM and the=20
ChildInit hook is run once per process not once per Perl interpreter.

Torsten

=2D-=20
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

mackenna@animalhead.com schrieb:
> Yes it is now Abend again. Timewise this is like relationships I
> have had with Philips (now NXP) European colleagues...

And a new morning again, and it goes round and round ...

> On Jan 18, 2009, at 10:56 AM, Michael Ludwig wrote:
>
>> mackenna@animalhead.com schrieb:
>>
>> Thanks for replying again. What you've outlined above is what I had
>> implemented following Mark's suggestion in his first reply to my
>> initial post. Unfortunately, this quite reliably produces SEGVs, so I
>> think this usage is not aligned with the Berkeley interface design. I
>> don't know in which way this is wrong - I guess I'll have to ask the
>> Berkeley folks in order to maybe clarify this.
>>
> I was using berkeleyDB 4.?, but without the fancy threading stuff you
> mention below.

The BerkeleyDB module is based on the C interface while Sleepycat::Db is
based on the C++ extension to the C interface, as far as I can tell.

> Since you can't share a tied variable (sharing uses the tying
> mechanism), while I was futzing with threading, I tried to share the
> underlying filehandle and use the function interface. But that gave
> lots of SEGVs so I quit sharing anything about the DBs between
> threads, which got rid of the SEGVs.

Good. So you got the BerkeleyDB module working on a threading worker MPM
by avoiding any attempt at inter-thread sharing.

I'm still getting SEGVs trying to do the same with Sleepycat::DbXml.

>>> What this means is that each thread must open the db's for itself.
>>> The amount of data stored for each open DB connection, times
>>> THREADS_PER_CHILD times the number of Apache children at any
>>> given point, makes for some memory. But
>>>
>>> 1) the separate connections help the DB package be thread-safe,
>>
>> So if coding the handler as above means that each thread, having its
>> own global variables, opens its own handle to the Berkeley
>> environment, I shouldn't need the DB_THREAD flag (which, according to
>> the Berkeley documentation, "[causes] the DbEnv handle returned by
>> DbEnv::open to be free-threaded; that is, concurrently usable by
>> multiple threads in the address space." This flag is needed if any
>> handle is used by more than one thread (or process) concurrently. So
>> it shouldn't be needed if the handler is coded as above and there
>> isn't any concurrent access elsewhere.
>>
> I have not worked with this flag but from your words it sounds right.
>
>>> 2) the first-used threads keep getting re-used in preference to
>>> threads not yet used.
>>
>> I noticed this is indeed what seems to happen - whether by chance or
>> as a feature, I don't know.
>
> You can find it described proudly as a memory-minimization feature in
> some of the Apache docs about worker and/or event.

Does anyone know where this is? Haven't found it here:

http://httpd.apache.org/docs/2.2/mod/worker.html

>>> 3) if you consider each thread as more or less equivalent to a
>>> child process in prefork, your total memory requirement is less.
>>
>> From perldoc perlthrtut: "In this model each thread runs in its own
>> Perl interpreter, and any data sharing between threads must be
>> explicit." This does not sound to me as if there is a significant
>> advantage over spawning child processes, at least not on UNIX.
>
> Start a prefork Apache2 (if you want, let it run a while to give the
> children a chance to grow. Note the (largest or average) size of
> child processes.
>
> Start a worker or event Apache, let it run similarly (or not at all if
> you're impatient. Divide the (largest or average) size of child
> processes by THREADS PER CHILD. My point was that this number will be
> much smaller than the process size with prefork.

Okay, I see. More concurrency bang for the memory buck, despite certain
restrictions, if my understanding is correct, of the Perl threading
implementation.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Torsten Foertsch schrieb:
> On Sun 18 Jan 2009, Michael Ludwig wrote:
>>
>> Hmm. Not sure what to make of this threaded Perl.
>
> In fact, it is worse than fork()ing off unix processes because the
> interpreter data is completely copied while a new interpreter is
> created. On the other hand a forked process copies only those pages
> that are written to.

Thanks. If I'm understanding correctly and this is true (and not
considering SEGVs), the total memory cost for a mod_perl on a worker
MPM should outweigh that for mod_perl on a prefork MPM, which seems
to not be aligned with what Craig has observed.

> I haven't used Sleepycat::DbXml but I use BerkeleyDB in production
> with prefork, see Apache2::Translation on CPAN.

Looks interesting. I'm going to take a look.

> What I have learned is, you can't open the BDB env in the parent
> apache. When I wrote that piece I also had problems when I opened the
> env in the parent and then closed it before forking children. Perhaps
> they are gone in a newer version.
>
> I think with prefork opening the env in a ChildInit handler should
> work. I ended up with the connect-at-first-usage approach
>
> unless( $connected ) {
> connect();
> $connected=1;
> }
>
> because I test almost all my modules also with a threaded MPM and the
> ChildInit hook is run once per process not once per Perl interpreter.

In order to be on the safe side as far as Berkeley DB is concerned,
you'd have to make sure up front that the environment is safe to use,
which is done by running recovery (DB_RECOVER flag, DB_RUNRECOVERY error
code) in a single thread or process that has exclusive access to the
environment.

Given that we're experiencing problems (SEGVs) when trying to open
the environment up front in the parent process, it looks like this
requirement is not easy to meet on the Apache/mod_perl platform.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

--Boundary-00=_Z3GdJJwjPdG45Jq
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On Mon 19 Jan 2009, Michael Ludwig wrote:
> > In fact, it is worse than fork()ing off unix processes because the
> > interpreter data is completely copied while a new interpreter is
> > created. On the other hand a forked process copies only those pages
> > that are written to.
>
> Thanks. If I'm understanding correctly and this is true (and not
> considering SEGVs), the total memory cost for a mod_perl on a worker
> MPM should outweigh that for mod_perl on a prefork MPM, which seems
> to not be aligned with what Craig has observed.

Of course it depends on your setup. If you configure only a small number
of interpreters then the overall memory footprint can be lower then
with prefork where each process runs its own interpreter. But the
effect of the copied interpreters outweighs this very soon.

Measuring memory consumption on Linux is not simple. You can't simply
add up all RSS or VSZ figures or something like this. I know of 2
approaches that work more or less well.

1) call mlockall in the parent and then use /proc/PID/smaps (or
Linux::Smaps) to measure shared memory

2) take a system with enough RAM and nothing else to do, start apache
with different numbers of children and watch the free memory with
vmstat while doing so. Then you can estimate the amount of memory that
is consumed by one apache process. See the attached picture for
example.

> In order to be on the safe side as far as Berkeley DB is concerned,
> you'd have to make sure up front that the environment is safe to use,
> which is done by running recovery (DB_RECOVER flag, DB_RUNRECOVERY
> error code) in a single thread or process that has exclusive access
> to the environment.

You can run recovery in a separate process spawned at startup by the
parent.

Torsten

--
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net

--Boundary-00=_Z3GdJJwjPdG45Jq
Content-Type: image/png;
name="experiment-scrambled.png"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="experiment-scrambled.png"

iVBORw0KGgoAAAANSUhEUgAABLAAAAOECAMAAACGszjIAAAAAXNSR0IArs4c 6QAAASlQTFRF////
AAAAoKCg/wAAAMAAAID/wAD/AO7uwEAA7u4AICDA/8AgAIBAoID/gEAA/4D/ AMBgAMDAAGCAwGCA
AIAAQP+AMGCAgGAAQEBAQIAAAACAgGAQgGBggGCAAADAAAD/AGAA47DAQMCA YKDAYMAAYMCggAAA
gACAYCCAYGBgICAgIEBAIECAYIAgYIBgYICAgIBAIIAggICAoKCgoNDgwCAg AICAwGAAgMDgwGDA
wIAAwIBg/0AA/0BAgMD//4Bg/4CAwKAAwMDAwP/A/wAA/wD//4CgwMCg/2Bg AP8A/4AA/6AAgODg
oODgoP8gwAAAwADAoCAgoCD/gCAAgCAggEAggECAgGDAgGD/gIAAwMAA/4BA /6BA/6Bg/6Bw/8DA
//8A//+A///AVCa8cwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9kB EwwGIOIHSKwAACAA
SURBVHja7Z17zDVHedhnzl6KhWq3OFyc0KpGqRM3SSUcQwmpA20VE0iFCYg2 lWosG6eFimKUCIFT
zLWt5QanSIaEQEJKVVARFzvFDrJdKBHGhlJDGww1GGiEuTTEMgSSP+rISXdn 77uz99vMzu9nv+e8
77k+33zv/r7nec7MjhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AABuIOUSDxnwzO5X
ab83vSd7QHSteWzXG00If/qfeMKbyE3eDcBGNzUOjhHC6nqo1B5284SVvGZN WOPNmPw8ygptD5Zy
Qbvkr4SwALYVlvalZ2Yt6WsOElb/Gy0krAX1grAAhggry1w0V6Wspu2R1Rvr dVv5IVmx0/90Udwl
m8LqeHp8KctvJGQt+K6Xbo2l7b5qSOWHVN6vcSV1j8xfGmEBaI2V/F9kLo2r UpeoeV9xeFWfIOqi
y+/LL7rerzBe9cbhATbeqBKxxgpdsXTfp/1jDh3PitrrAQKAXljqsCsdQKVG dnFf4yGiyJe0ne/y
8/L8onKIlu7TNpo0dtBYU/YKq8ip5ghLE2dDOhWJ63pulXCLQCXCAuivCbX/ 1AutvtqF1dJ4aTyv
RVjapw/NsDT6EqLs0lIxVrFM3Tz6TKknTr2w6m/QvLs7R0VYAK1drHZF9ZVv lXKreSSX+02DSsL5
wmozo74k7K1g253dJawBQ9f1SIQF0CWseqJV/knTiq8c+rL+dG3yJvQ95mqj u/7sztpM6EvCQW9U
/QSgZSRqMmuNs5J2DWy6N0pl7Y0AsIsQt32/huEAAEwUlmhOSOAvAQAAAAAA AAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAABxFAgDMZithWeRWQawMK6GaGSrC4sgiVIYVYfErQKwM K6EiLEJlWIkVYTGu
HFmEyrAiLH4FiJVhJVSERagMK7EiLMaVI4thJVSExa8Aw0qsdoV6jea2V6vL r341vvyC+Jz66Z7k
rlN88eX44iXpg/+OurzkzU+6KQ714/8LYS3PSRCr48MKKW+q/fyp6Ouy9Puf iYX1hc+9J7vvTvV3
fL3ylbgt+npULizx5uTq42RYCItQJ/0+ymH3DVoKsuh6kc4XW3xlSrq3dvfr vir6eqF4fiysT+U3
vTv55nPviUQlro2+nnpn6e/4j5Or31Dp19e+Vghr7p8AYYGLJMeN/ujRCWu4 RrSP7Paj5vldkWlX
1PWIIBFT65+m89lPjC+e+/xYWOL8q9RNr3xV5REXKGHF371bXB1d3hB93Rt9 /V/xgtdHwjpneKAe
wgKYnrAMElb/a3Y8Pc1xBj1B3TrmCSUplp4nRwnrPHFx8s2XhDhbxMK6QLwy /vlf/158efb90cVf
jb/7r8nDPiFuuEE87t5IWGcmWdj3PSV9pXMRFrjnGCnybCErZtLcQXdVXeFf eoJo3idL91WfXjtP
QP7uWebSeIL+6S0hlROtamT1ty09ofTumneoZ31lYZVfNH+0PsCnxC0reb2U D4p3xzeeLR4r8z/u
S39PyKvl2ff/7+inq2KRnaXe5BPR3498nPjdT14h3y3Sl09e+tx+tSIseliH CrV0OGbHmWzcqLkq
S0nqC6/SAd14I+0jtU8ovXLj6dqQavarRNZ4ZFNYLe+g0XRye81mPWP2FPWQ 62X63D+/MRJWmq69
VZwR3XeXEDdnr3uvlC/KbfbOdypRRf9dKd77c/LC7N27hOUlICyEdTxhpaYq X5UPZVGT2SRhNTRU
exUp63Zpe/fGjY0/lTYyqUvG9JF1CKu4UWfwln8AysF58Y03xDde/ojou0eL t10sP/vWXzjjy2+X
d90aN7bU0264Vz70ot+Uv/QxcdW34yf8yU+86cork9f8Ofmj8S3PEGeSYYFz JaHUZRmV+qr5EK2U
pgurUsVVG10dVqm4pSEsjZXqj9RF1njbNkt3y6wcROltb1LCEr8s/4O8WcRz rNR9b7v4Qx8Sr1JP
uDU7um/IXuwJN8qPS/lt8Sevu/YzyWveVi216WGBQ77qrWS6S8L68d12X7Pi ajvKmyVh890bN/bU
l0KXDWkj63zbhhIbRmods4wHxF985fpIWNWnvkzelD/ynE9EN/zN6OvWRIu/ q17lq0L++COib35f
yttuUy/76R+S8ucRFriYYTXb3u1Nd+3xWW2w6+5rtJbamu7NAlGKlqdX0sOW P1Gj+a5JscqN+c63
rf3hNJ95lorqtgC/ItLeubjrir+RPO53xE3RN8/0PPmyN+TCih5y3tfvTV/y ASFuf8Qj4tvlGfK5
UXb2vXLFa75I6GHRw1qwhwXb8JX04I2G/NnxBKsPPu9H/kDddHXyd/CKt0YX T1ff/tYLlLC+/iTx
Z/LOKO9KLPdr4qfjR978HPE9ce63LBIJwkJYS2ZYMJpkXWB1rcyrK4/48J/H l/eUbvnL8cVLKg+6
5KlPSpT1pmLdzgvzf00uK/9VfSH7m3rh/xBfUr8AybqdZBoXwgKAHkrCek11 cV+QCkuIJ59fEVbK
nen1U5Orq9Vc9hy1CEd84AOVNztPfFH8GyW07/9S+XaEBQD93BV9/b/0+9dE 33z8onh6ejIh/QYl
rHgBzpOf3CKsa68Vb4yMdba4KL4hFla8ZvBapS9xv7jnHvEB8bAQZ+XPuSkS VmSsKzJhqTnvF6Rz
3hEWAFQL6MZN1wvxoLr1ghvFYy9S5smEFTlI3fOb6ucr1R1/qDQjLzvrna9V 971R3XfR34/ys/TF
z0sSt/vPVj89rIR1eSas+OIMccWZd4lUWFL+pPhFce4nEdZi0MNyflgPJqzU LMpDMhLWu6RS1I1K
SzeWrXazVF0tLxWWajrdql7hrLM+9En5C1kTShWUUt6Qfv+E9Prbmb/UOz1a Xb1VfbwYC+tnVEH5
wZ+8/BfvLR5Vxi9dIiyERahOa+tFyQ8fuzoSljrZwhsfVMKS71W3Xxh9PSM/ PZ8Qr1M2eou47bbk
BS7+kPj56Ia3i1tjg/2Retn4IVckwnr4ZeKm67797dh118YLoOO77xA/mh77 T7vpAfEX96tTZ30w
DuYMrbCUq3wyLAD7HLPWGm/xQ9ENX0nmKjw2+vHl6unniguTx/6O+Gh0+cyv /vbrXifl03//LW85
T/zqC6K7zvv6vcWU1tvj6Q7y7McL8Vx5RWrEC6W47rrkHZ59jpT/TMgfEfJl ovS5bnT50Hsfin74
LXHGV0WLsXxKQgAbc6IV1nj/n1RYIl9c+Kep2c4VT5Dy3G9J+cOPEuL556n7 f1w+/f2Pk+f9qnhB
9JCvi1hYT0he+nb19Me/Nr56jvxe9u5X/Ufx39RLPzv+6Xm3JIf9t5R6XyHv id/vvT2z6HyBsACs
Fdbya7zFnz0QXZwj/4X4S1L+4a/JX/npqH6TX3zi16R83HeF/OG4DXWektk7 pHj/Gx5bHLyXRtf/
XMrvfVrKfxrd/TIpPxq/8mXy6sfcEj3hm+JfiuTjQykviS9vUe/8rfgxd0ff 3XNWGtKzu3xFhkUP
i1CtLAnXWOP95Ysf/cflZdJSPOtXLhF/9J4vJu/33Y/Fl2fkp3aQ8g3iISH/ evyEX1fCiowlxNn/
PhbWD2TCEuIx8l3/vVKQfkO9+jvSd/kJdeNLzjpLilsSYV0q6GEhLEI9VIYl 1ljjffG/ukT+snjz
lW9745lnRrc9ST4rveMfRD/Fna1zxBnRg/+LkM2z4chXCnFl8tJeuW921Y8J +S6hjCXvVg/8xjfU
895xjZB3JMH8lJT/85bou1vii7936aUd9SCfEgLYmWEtu8b7wXjawXUyElZ8 0z9RB+Vl8pvffE6u
yPQJz8tL0MRbD2ever4Qb49veTh75D+KH3WVeJQ8U3wquuMZH74oDfO++Ooa 8dKX3qF+viK6+MH7
0pCEvNR8kSAsgLE9rGX5d/HFdeLfireKZwn5iU8kt35TCGUs8frzU0W9LZ7N oHifKgXj9Pgx6hA+
X6qrhx/+2eSIjreneODxQvxnFfMlQnz4onhy1zNiYT1RyAcvqRz39933mUQF 8h+Kv4WwAA6XYS3P
9ery7md9RH4wE9Y92XLnl3/60+l3b3ldfG5Q8YAS1jVJPZ/M44qnXt0ufjb+ 9seSx17+mVhYfzf5
4Zbk6rfFVeLu9LVebaVI6GG5His9LCO4Xnw2qg3v/sjb059vV5cPPhjPHH35 dZ8X4pnRj3eIC87M
nvDEm6/53h/kT49PO/qa219TfsXLxYvFnbGw3hn98P74ZA8fyGUW8eJcWP9Y iEeqb/42wkJYhAp9
/KC6/Gw8wV18t3HvhUI87fPJt3eIn0pNdo1SVCKsh9IHvkZc+AO/7t+svo93 eH7xi4V4T+SrWFji
o4mwEn7pfcn1d8RHPqJOaPPI+8mwAGYUXun1gi/Wfqv2FO1j3mBasPkTEmH9 lfTnV1QfFSc/T8t+
+L5MWFH+9P3itfmD3pAdyH4srEfGyw3/U/T1pvvyR0T51Hf+2tmZsOKLPxXv +44qKONmvxLW61//
+RciLIDROuhwwOi9T9tepTF9Sv+E2k6Emk2e5wpr+KawHR20NitvdVpFhAUI a8Jdw4U1LKLGnhLN
TZ6XEtagJ7Q9uHVj642MhbCsbrbQw2omBtXNjPV7GjfyGCFGbtbc3HRZVF+l bT9DUdvkWSssKVqE
VdowovKabVtFNwek5kLdVtHa7aBFxxu17x/dNnsMYSEs50Pt2gu5VJ4199Tq 2Ou5ZbPmfB8ZzYvV
ly23BthWIla2DqvuIdgxR7Rzf6/K22k3C2vfrFA7dJq1ixOeFwwCYcFBa8Da P+nVekj2CquyEqVr
s+ZMWLWlK7LxRi1b3Evdlleylu6l31e1IDqEJTR728uSNfXCqqyrrgxkOQmT jdVCdcOJli1etUu2
ybAAahsy67Zg7suwGh0f/WGmPwTb8zStXPQNLp1VpgqrluY1VSx6nteyK7R2 M1jRvd21WEhZCAuO
JCztXsii2djqLgmrtVQzw9I+YZSwGhlWl/26S8Ih4ikKTH2vratZNnCxtWg5 uURfzbqRSDrWYWrb
a/Sw6GFtUhPWd1tuloTlR9bOM1U5nDr2QtYnKbqmfUfTvaUM04Sk9UhbYlZv usvGgNQy0M6+WMWJ
moXYHUEY1XQff8ILjiyEZU7VeLw62JXBQljgYJuLP5FjwipP35CdV5WMN+YU /1Nr8oUwPD63L8DR
f2kyZmZYsu+KDAsAKAnpCwl6WPP/3V6u+DF/8fMSpd7edSLCQlgHDZXFz7W3 cXrx84GnNcBR2h6z
HsPi5zHCYvEzwLR/Ra1e/JxlZSx+RljgQEpl/eJnWV1LeOzFz+EgEBY9rEOG eozFz9WzNbD4mQwL
YR1WWI0D277FzzVhsfgZYYEzJaGdi5/bhcXiZ4QFRzKW9Yuf9Q1rnUf0aQuL nxEWHCsJ409k82Ah
LHpYDoWKsBAWwiJWhpVQNwoVYQEgLISFsACAkhAAEBbCsjDJpnYBhIWwiJVh JVQzQ0VYAAgLYSEs
AKAkBACEhbAsTLKpXQBhISxiZVgJ1cxQERYAwkJYCAsAKAkBAGEhLAuTbGoX QFgIi1gZVkI1M1SE
BYCwEBbCAgBKQgBAWAjLwiSb2gUQFsIiVoaVUM0MFWEBICyEhbAAgJIQABAW wrIwyaZ2AYSFsIiV
YSVUM0NFWAAIC2EhLACgJAQAhIWwLEyyqV0AYSEsYmVYCdXMUBEWAMJCWAgL ACgJAQBhISwLk2xq
F0BYCItYGVZCNTNUhAWAsBAWwgKAZQlDhAUAlviKDMvuJJvaBdzyFcJCWITK sNoQahhuKRJKQsAC
hDovvUJYAGBLOYiwAMAGXYVbi4QeFgUBwLz0CmEhLEJlWA0PNQx3EAklIWAB Qp2XXiEsADC6Ggz3
EQnCAoCZukJYdifZ1C5waF/tJxKEhbCIlVDnpVcICwBhGRCqNyi9QlgAYADe oPQKYQGAgcIK9xYJ
PSwKAgC9rRL60yuEhbAIlWHdO9SasDp0hbAAEJZRwgqNEAnCAoBeYYWhGSJB WADQJ6zQFJHQw6Ig
AOgWVtjrK4SFsAiVYd031ExY4YAXQVgACMsAYYVDfIWwAGD/kjAMPaNEgrAA oE1YYWXiKMI6aJJN
7QJHEFYYeggLYREqw2p2qKmjwvrSHIQFgLDMFFYYeggLAMyvBYWaKoqwAMB0 WyWTGQTCcifJpnYB
q52VzL1CWAiLUBlW80MNy7kWwgJAWMaGmky+8hAWAJhPVA56ZFgAYIWvKpry hj4NYdmcZFO7gLXp
VUVTCAthESrDamqo1ckMlIQACMs0mrtM0MMCALOFFeomMwwsCYMAYQHAdsIK mwobnGEFZFh2J9nU
LmCLrRqnFR3dwwoCSkKERagM63YZVqgtEgcJS+lqskikIrkW3VeUhGAbCGuV DKtxGuQR0xqC+SKJ
dJQ4q/MKYQGAdg+vwcIKgvkiyYSEsACgV1easm+osAKxQKmWe0l2XlVqyJhT nMRysczFyZ5wTxYN
rOCXa+GLMLrwqrelnav+5wbBqeSPOb7qtZWVGRbdYYaVUJeuBrWN9UGzr4Kq d2YKi5IQjgfCWqN5
1ZwjOmTqVSAWEVaiIoQFAFoyG4W1G8YIq66rucJiWgMAdAiruRDHG/4SgUY8 G0VPD4vaBdwTVn0h
zph1zkGwo0gQFsIiVndCbVuI4w3PsIJdRUJJCFjAqVC9ULMQZ7CwgmBfkSAs AIfqwWSLCU3W5U2Y
y4CwAGBNGusGxwgrCHYXCT0sahdwyFfaeaIjTiODsBAWoTKsW6VXQtOrOolh U0WNEAklIWABR0IN
9cnUaUiGFQRmiARhAbhRDVZX4tSqwlnpFcICgEWobjGhS6b6m+5Br68QltX1 ALULmCWssCOZ6hVW
MOBtEBbCIlSGdRlhhWFX9dezkDAY4iuEBYCwZtsqWYnTyKaGp1fDdIWwAGAR Z4X6LXGGZVjB0LdB
WAAwP8NqrsTRCWteeoWwLK8HqF3AEMJ+P7WXgwaKBGEhLGI9bKjaPXEGhhqM 8RXCAkBYc30lhiy6
8WanVwgLAOb7atCyZm92eoWwAGAKXqkc7MifuoU1WlcIy+56gNoF9haW0lV/ hqV7RDDhfREWwiJU
hnWqsMKwq+DrzLCCKb5CWAAIa6ytsnQp7MifeoQVTHtrhAUAkzKs7qWDncIK JvoKYQHAlAyrthTH
G/MCweS3Rlg21wPULrCXs8LOgq8zwwqm+wphISxCZVhHZ1hhOHT/5oaw5ugK YQEgrNGEwhv/JG9m
NYiwAGCCrkIxWljp54jBTF8hLAAYm14N38K5mmEFs98cYdlcD1C7wDZMWInT fI1gvq8QFsIiVIZ1
sLDCgStxNCyhK4QFgLAGCyvUpFxDdTWlikRYADDWVmlCFY5aiVP11bQqEmEB wLQMK9TViEOrwWl9
eoR1rHqA2gW2yrCmr8TJmldkWBxZCIth3cZZ41filKtBQYYFgLC2CXXaSpxq ekWGBQCbEIaTZRNU
ykoyLABY21eTZVOdekWG5Xw9QO0C6xSBlfRqomzqM0URFkcWwmJY1xVWOLmc q+qKkhAAYa0srGKq
6Nz0igwLAFaxVZ4MhZOzo6C7zkRYALBwhhVOlY1+nTPCcr4eoHaB1YQ1ck+c Pl0hLI4shEWsK4Sa
FIBTV+K0n0UGYQEgrHUyrHDaE1t1xaeEALCKrZKlOEvqigwLANYinLhusC9t Q1iO1wPULrCCrsJJ
1VvvKZARFkcWwmJYF/fVNLf0n7EdYQEgrAWoLh2c0h8fssMEwgKAJYWVzb1a fkMcPiUEgGWFFU5L
hoJx74KwnK0H6GHBfFtluU84beng4P0GERZHFsJiWJfKsCauxAnGvQvCAkBY szOscNomXmO2c15k
VBEWgPNVYX0lzlBhBZuHirAA3LbVSitxKAmpB9yuXWAdwmkrcYKx6RXC4shC WAzrbF9NmyE1vhpE
WAAIa256NUkl49MrJo4CwFxfTVPJpGY7GRYAzE2vxBorcRAW9YDjtQsshVdK r6aoJJj7vggLYREq
wzpSHGuvxFlnVBEWgIPCmrgnThDsPKoIC8ClenDWnjjB7vEjLADHnFVNr4YL K9jfVwjL6nqAHhZM
yLCmLcSZnV7RdOfIQlgM61jCaVsOzk+vEBYAwhppjWlb4izRvUJYADA6vZqy Jc789IqlOQAwVldr
bYlDhkU94HbtAsv7apo4lvpsEGFxZCEshnWoJ/Jm+y7pFcICQFjDhTVtIc6S U0WZ6Q4Ag4RVnssw
ZkucwLQ/DcICOLCtNCtxdkmvEBb1gKCHBUMyrHDaysGl0yt6WBxZCIth7c2w NtkSB2EBIKz5oU5d
ibNGqoewAKBLEtNW4qyQXjHTHQD606spK3HWkScZluv1AD0s6PTVnitxEBZH FsIi1uGhpt2rvVbi
ICwAhNUZqldJr6Z4YsWpoggLAPROyNKrrbbEMV1YMmLYFcIC2FhYU1fibGTT 7YUVqyj6GnBln7Bo
tjg/rJbaKs+mpq3EWTe92ndaQ2oihEWsDKtBoTa3HByxEme7enUPYcncS7Lz qvwExSkeYpMvhOHx
ccFFcXEqfswWOk96qSBYPVxvxnMLf8wsCWXfFT0sgO16WOasxKGHBQDdTthr C6/NQFg0Wwj1ILby
wnDSDl6BRX9IpjUgLEI9xrDGe+IYWg0uN6pMHAU4hLDCSTMHNkyvEBYAFOnV lIXOW5asCAsAlK8m
OWHb7hXCcr4eoIflNtWVOKMzrGCfaBEWwiJUJ2P1klDDSU7YOL3ijKMACOs0 caHzDnMZyLAAHK4H
k4U4Exc671jAIiwAN50lbJrZjrDcrgcsi5Wm+/IZVjhxy0GL/9QIC2ERqpWx eskWXpakV4KJowAu
CytZiTM61MDuUUVYADZi/EocSkIAyHylqkJbqkGE5Xo9YF+sNN2XLQcnCCs4 wJ8cYSEsQrUt1kkr
cXZPr+hhATgkLK/iq/SWwaHuP/UKYQE4hFcpB8dlWDulV97ir4iwAGwSVjjB B7tVgwjL2XrA+lhp
us879KcvHdxLV0udoAFhISxCtS3W1FeNBKY/1D177fSwAFyVq6fbc7Av1H0/ G0RYAI6WhFP28Np7
KgM9LAA3mbKHV3C8YUBYltQD1sdK032WrsLxHezggL5CWAiLUM2PtX3l4Gl3 XXmbjirCAjBcWHGz
vW2OwGn37AphAUBzJY6hywa9TUcFYQGYLKzsxAzDe1hb6mqVyaEIy+Z64Cix 0nSfIqwJe3gFe8SJ
sDiyEJbDsXqame2Dmu7bfzRIDwsAucZ7eIX9Zjjtm14hLABQIhi/5+AhZ14h LADjbTVhJY4DukJY
ptYDx4uVpvsoxq/ECZwYF4SFsAjVuFiHr8Q5rZ9eeSaNKsICMExY4Yg9vE7r p1cICwA6qsGxEzLX
7V55Jg0OwgIwAa+SXo3yxKrp1dZT2RGWZfXAUWOl6T5IWOO3xFkpvfLIsDiy EBaxdgpi/Eqc1arB
5YVFDwvgGMIavBKn4avVQkVYANAhiCErcbZKrwzrXCEsAJNslcxsH/ksdz4b RFgm1QNuxErTvZNw
tCBWXomDsDiyEBaxtugqHL/FxMqhemaOKsIC2FtY4WhB5OmVa/9kISyAnavB EStxNqkGTQZhAezr
KzF6JY7Do4Wwdq8HHImVpnuF6XviBC77CmEhLELdI9bpe+LMDtWzeVQRFsBu wpq/EgdhISyAlW2V
JlThWI0sUg16Ng8dwgLYw1mithRno5U45i66QVjm1gMuxkrTvWaNsXPbF2u2 k2EhLGJlWMcRjt3E
K1gsVHpYlIRwKNYX1mLpFcJCWABrp1cjG0kBY4awAHbSlWAlDsKyqB5wMlaa 7pmvWImDsBAWoRod
q1dJr8RyK3E4WwMlIbjOesIauxKntxo8deoRYSEsgKnCKs1lGLgSZ2Y+R0mI sADG+mPiSpzJzXaE
5ZKw6GE5P6yrZFjjV+LM0yPCQljEyrBOVEi40koceliUhOA6y8t17EKcwekV wkJYAMtmWMnM9uXT
q44C9KggLICVCUcqhJntCGvXeoBYhcNN93Th4PAMC10hLIRFqPvEmp20fa30 ipnulITgOrMPLa+i
qzEZVrCHBRAWwgKXyfbEaRhsTnrlMa4IC2AtYYXj98QZlLYhLIRlaZJND8tI WyWnbB87sb03vUJY
CAthEeoasXoTZooGAyzoOf8bgLAAlj20koU4Y1fi9BeZHv9kISyA5Rm7bnBq ekVJiLAA5taDY7eY
GDKXgR4WwlozcyVWm4d1Xnq1ykocbIWwEBahLh9rKFZZiePxG4CwAJY+tFZe iYOwEBbATKp74ozI
sDgvA8IC2EtYo/fEYegQlhmZK7HaPKzThDV2T5xqekVXHWEhLEJdP9bJe+K0 lJX8BiAsgDUPrQl7
4jS6VwgLYQFsUA9O2hNHn6UBwgJYm7ErnXUfDmKrNUUiFcm16L6yTlj0sJwf 1nEZ1tg9cfRzGRDW
usLKr6KvriuERazHHtbRK3FavMdvwNoZ1kGFBc4z4tAauyfO0lNFEdbgDCuR kuy5qtSQMac4bpMv
hOHxcWHQr0AoTrmwhjwjYHQnXRT+mOG6fluRYcEBm1al9GpcOcdKnL0yrFxY 9LCI9XjDOkxYIStx
rBHWAFMhLGI95rA29vAakGF5a6VX9LCGGqt3PoOt0xrAeToOrXwhzriVON5a 1SDCMq70BDCoHkw3
8dIWiVSDhxEJwoLDOKu2cHBIesXCG4RldOZKrDYPa0+GNXIPr7gaxFYIiyML YW0fq6/SK7/97uKy
0NWKwqKHRUkIrnNq11EYer7ntwvLzy7KukJYCAtge/zQ92Nf+W0+8xNqugJK QoDNCUO/aqRaFehX
hIWuEJY1mSux2jysbb4S5RTKb1SBua98dIWwOLIQ1i6xVlbiFEYqpOXHovKy bzJfMayrhIqwAIYI
K6zWfH6eWKU/l2zm+9ulVwgLYQHUhRWmvipJqfqTVxYW1SAlIcAetkoW4hQz 4Co3fgAAH+dJREFU
28u+8oWX2ir6PyJPr3wGDmHZlbkSq83DWsuwygtxfL+cY3m5u/zszqga9BEW wuLIQlg7xJps4eVV
hJV9+b5XzrZSX2UNLYZ1lVARFkDXoVXbcdD3q1lWnmupm4OtfYWwEBZAoauw tqomn2+VfWIoiqrQ
D9LikJKQkhBgB1/pz39cclSiKPV/OpcBWyEs+zJXYrV5WHNfxRfNdct5XpXN IRXMZUBYHFkIa99Y
s00mNMIq14NJjbjXShx6WJSE4DqnIr3SlYTFukGRZlj7zWVAWAgLoLQpjvZU VslMhrQeFEG+uBB2
F8m83VYRFtiFVy4HW4WV+SoWlZrZzmeDhghr7v7Q9gmLHpbToXrV9Kpll9Ts M8I4vcIiRglrGd0g
LIRljbCqm+LoS8K0h7X7aa/oYTkvLHC4Hkw2xTmJ3gwrbbgHh7DAYYQlMxAW OEJjz8GWDGvb014B
wgLQ6MoTQ4XFaWSME5Z97+NWkk0Pa2lfaSrA5lrCZBZWwDwGE4VF051YHQi1 OK2o1x1rembk5LwM
PsO6dagDpjUkX5SEcGBiTYV5j71jI9V0SU6QL8tBWGYJi4mjcHRTqaus2d65 R3MpvRLMbTeyJFwk
v0JYYLCwkrkM2Xe6WQy5r9LTXpV3TwVzhOXkp4T0sFwKdeBKnFRYyWmvyKwQ FkcWwtrBVvmmOF7l
hlN7OZgsHPQZ1p1CZVoDOFwPJr6q5FWtnxKqlYNBdZt6hIWwADYqBJPmVdg1 7aoirB32mACEBW5b
qvpt2PGoajmYplc0sBDWATNXYjUvVI2wNCtxWtKr+LwMPpMZEBZHFsJaT0za 24u5C2HLg0/5aZBz
Qan0yrQGlqCHhbDgQKVf6cbKDKtiJU7boZVumJruOCHSlTi+cekVwkJY4IKw WnWVnu8429I5NlQQ
ZN9RD1ISAixoq2YmpRGZJ1qzq/pm9OokyMUmEwjLdGExcZRYLQp1mLDaq8H6 NqnqLH0+srJGWMx0
J1YLheV1CivsfAm/kl8Faeud3wAjQpWzH0BJCOaVhF0rmcOw5zUq6VXWZ/cP bAGEhbBgd2EJff+9
T1cVX/k+bXZKQoC1hZUmWbN8FeArhOVI5kqsO4Xak2H1loP5loOprjCWZcKy 7H0QFsLqOAdfOOAV
UmEFReOdYTUnVLngoygJwZiScJqv/KwmzNIrgbAQFsDawtLqakB+lZ20vUiu KAktKgmlpOkOBxFW
2JtaJatyfJF3r5guirAcyVyJdadQvZa1hJ1z20W6v0RSEJZ0hazsEpZgI1Vi tUxY+gyrO71KC0H1
fal7xbCaFyo9LDh+SdiRXmWuSn2VLBy0aD4DwtopkwOYVQi2p1edzXa/2M05 Pu2VKH6iHLSyJERY
YI2whK6H1fPZYDZPNEmvcmHRwEJYLmWuxLppqO3CCgcvxYlPe1X5GWEhLCSA sJa3VcdinCFTr9LT
yARVgdF0NzJUhAXW14PtU68G+Uqpqdh/vjh/H8JCWACbCWvgykG1AX3Rskpm j1ISUhICrOWsielV
oqkg8DVVIh8TIix3Mldi3S7UsVOv6gSatMv8Vc8Ia1XVICyEtaWwRugq0NWJ 9LAMDXXACfwoCcHs
erAhrFnpFcKyW1jOlYRgobMWTK9yYzGsNpaECAvsyrDmpleCprvVwuL0MsRq fKjepGqwzVc03REW
wkJYG2RYi+jKKmHRw9pJjADTs6s0wxquKxF03almjSIsO4UVn3UUYYG5wsoy rIXSq0RZzHO3tyQU
nNMdjM+wwqXSK7syLITVeIDkFMnEamqoRXo1Qld9vqLpjrAQFsJaMcNaMr2K Y7XnjKP0sJolIdt8
gbH1oGLR9EqkJ2wQNN1tFJZYxFcIC1YTVhh6i+oq2/OLktDGktCy9wEnNFUS VqQrb6iwgqFv4AtO
6W6rsJg4SqymhVoSVCiWTq/IsKwWFjPdidVcYcXN9sXTq7jpbo2v6GGtYxpK QljKVqWVOLV0a5H0
yq4MC2EhLDDeWXl6tXg1mAiLiaN2loQyA2GBaRnWWrryOb0MwqKHRUGwcIY1 2FdjdJXuW88J/CwV
ln3vg7COHmo6VXSor0boKpnLUOxKSA/LwFAHbULBxFEwpyKMv4anVyNklexR mBqLpTmWCisuB+lh
gVHCWiu9Sj8bTMpBakIrS0K5jG4QFixVDw5eijMyvfLLFaFgqjvCcihzJda1 QvXWS6+y/Ir0ymZh
8SkhsRoT6vClg0N15aeXSdcq/e9kT3pFD0tnrJY78ju7rigJYbmSMNTsSj+n Gsw/G8xSK/Uzpxmz
WFgdIss68l1XCAsWY+jSwQGnbK9kV9npRakD7S8J2x4lERZsritvkLAGpFfl prrP2WSOL6zESOr/
7quS3xJOcWbIxTIXJ3vCPc19gbBY9dz5uCDoeSlfXfh+/mPynV88RPDLZdiF HLboJpGO5lF5FtUr
LfsyLJruBoYaT22vbUk/qRosJoj6mlzL1cbQYXpYHcKiJIQty8HUW3ObV/kE Ub/Wx0JYRxCW6JqH
hbBgs+6VN0hYwzaYyOdaMbDu9LAE0xpgM1/ltaA3K73Ksyk+FDy4sOx5H7eS bCcKAnVihv7PBoPB
nw2SXh1aWFI6drYGhGVSqOl5ZHqFFfTnVZV6sNNY9LCMDXXI0hzO1gDbk54G ufLTuPSq5Cjha+yF
BQ4pLOe2qgcjbDVmB/qgqwIUpf0k2Lvr8CUhwoK9hDVUV0FbEejXMiqaVw4I i7M1EOvmoY447ZWm
GixOEONX8qph9SBYLSyxiK8QFsIaI6vYVzOqQT//KNCflFfRwzI2VKY1gHnl oJiRXqXGqvevfCzg
iLAWSbAQFowQ1uDTyLTdkZ7vmArQvZJQOlcSwp62UtWgNyq9aorJZ99md4Ul BBNHiXWrUJNm+zBf
lfRUPQk7Z2VHWAiLWLcRVjjwHMhBe0I1uxakh2VsqAOnNcyuCikJYQjJWUVH 6Urki5mxAMJCWLCl
r8SYcyDXz74AlIS2vQ/YiFfy1YB1zkG59vPpsSOsyv1MHCXWlUP1snJwCOVq kJ1PEVatIBQszSHW
TYQ1UFdBuRYUvlhjvhU9LGNDlb13L/MxISUhtNlqzFKcQNRc5WMBhIWwYCNZ ZcIaVg4WU0Wz/5jJ
TklYLQmre0kgLFhaWPHVUF35DV9hLIRVNZZg4iixrhXqmG67al5l/XXOxIew DvE+CMueULONUQfN
FU2qQV/NYCjnVwyrW6EiLNi3JByeXqXG8qsVIRZAWAgLtrDV4NOKBkH+kWD5 TFdASYiwYEOGz2XI
J7X7Pt12hIWwrEyyLS8IhqZXIv9gMOthAcJCWAhr21CLHeh7dOWXjCW2WIVD D8vYUBEW7FQNDjoz
Q5AXf2mGtUktiLAQFkcoVHylLgdt6OwLJrYDwoK906u+kjAoFIWxAGFZn2Rb WhAUU6+8vvQqLwJx
FSAshLVHqJmuujOsoNRg3zq7oodlbKgIC3aoBrPsyutKr5ITie6xwSDCQlgc qVBOrzqFlZ5GppjV
DoCwYL/0qlNYQTbfaqNZDICwLBYWPazVQi2lV+39q6zZ7rO3BCAshLVbqLXz MrSVg5X0ah9j0cMy
NlSEBdtXg8pWXelVXg4KhEWoCAu295UQA4SVThWlHASEBaakV6I9vfJLawYx FiCsQyXZtsQaVkPV
d9yzhYN+mmHx8SAgLIS1V3p16siw/LwaFNudQ+YYvwL0sCgJYfH0qoFXtZVf 3sJr/9lXCAthceA6
qiv9WZC9an4V+LUtBwEQFhiiq1qGFQSFo+hdAcI6apJteqyhPtRq090PijqQ 7SUAYSEsE9IrbdPd
V1t4+elmzgwroSIsMCC9ai0Jk9PImDaLAWEhLA5gx3TV4Ss//z/I54lSBQLC AiPTKz/dwTnbcZBl
OICwXEiyjY017A41UVQ2s51lOICwEJZJ5aBXCVUVgcUWXuadpY8elrGhIizY IL3y6jWhH+xztnYs
gLAQFvT4SrMSx7CJDEBJiLCc1FXTV7WTM/hGTr0ChGWpsOhhLZxeiVoPK0im MWArQFgIy7D0qiGs
IBDG+4oelrGhIixYXVflHlZgw+n5EBbC4nh2V1f1lTjMugKEBfv6qvNeL9dV sS4HAGG5lWQbE2vY
7SvVw0r3nxf02wFhISxzdaUmNRS6smAlDj0sY0NFWLBuNVgUgwiLUBEWGF4N Kl9VluZQEgLCApPT
q+auXgAIy7Uke/dYB6VXojbTnQwLEBbCMlRXqntVFpYF++LQwzI2VIQF61aD IhVWtR6k6U6oCAsM
rAYbZ2sAQFhgbnolGifwA0BYDibZu8Xak16pgi8olX2eXcMKCAsJHCjW3vQq 2X7eWmHRwzI2VIQF
C6dXSTlYbatbVhIiLITFce6Srmq+oocFlIRgUDWYTK7y02qwMTkUYQHCIsne PNbW9Cpf0pzsj1qZ
HMq0BkBYCGuHWDuqQXUeUV/42Un6RGVyKE13QqWHBSakV5mc1Ina/UD/VJru hIqwYO/0yhd5IaiE
FQS6hc2UhEBJCPv7Ku9cKV+JUjVoeYYFCIsk2+pYW8rBpHOlfBV07Y/KTHdA WAhrs1jbJjMkxJoK
8ozrACUhPSxjQ0VYMNlXubCydc6tp+aj6U6oCAv2LAdLwsomtrefmo8eFlAS wq7pVS6sYr9B0WYs
hAUIiyR7g1g7lw4m6ZV/uGEFhIUErIy157xXka9832dYCXWrUBEWTEyvYmGp uQzH2wMHYSEsjv7D
6Sqe2J4uyQEwWyQyYtgVwrLVV30PCJJJ7hph0WMHs4QVqyj6GnBln7BotgxM r/KZ7oOERdMd9i3V
ENaBYx2QXonjCoselrGhzisJlZdk51Xp8QmnOG6TL4Th8W1wodKrrocEgfou n+levjdfieP4IG5z
cbIn3NOM5xb+mJlhyb4reliH7F5l6Fvu9LCAkhA20tWQ7lXhK52xEBaYJSya 7keNdYyuaLqDLRnW
gac1OC2sEdWgKG09wbAS6jahMnEUJqZXonO5s9UgLISFDSzw1bD06qiaggOX hAjL4fTKx1eAsEiy
d4114FRR1VH3J/iKpjsgLIS1cXrl+5GwJq13pulOqAgLNq4GI1l5ybYTRwZh ISysYLCvBlWDCi/W
le/RwwJKQjA1vSp/LpjVgxgLEBZJ9uaxDkqvih1SVUl49GEFhIWwjIx12FlF S3lV0nRnWAl1n1AR
ltvlYM/9aguv8oJBN8pBhIWwkIOBuurZE0d1r3yRbUcPQEkIhuoqnctQnJAB YwHCIsneK9b+Znvm
quykV74zwwoIC2EZFWtrepWeyspP95jw/VHnkOHfAUJdN1SE5WY52HqPl6RS efMqc5VTBSHCQlhI
wiBddZSDXiW9onsFlIRgbHqVEGSNK/Z1BoRFkr1vrN3ddi/+cHDR5pWFwwoI C2GZEWvnZAZfzWXw
04ntyzavaLoTKsKCqemVdicu3w88sdhEBntBWAgLVxjlK42w/GThYDH5iuEC SkLYUVelctDTpVfl
WhBhAcIiyd4x1lJ6lVC9Wy0cTISVZVyuDisgLIS1c6y1bnsjwwqyqQw+w0oP y9hQEZYr5WDt50Z6
la3EoRZEWAgLYe2sq8ZkBq+uKyGY2A6UhAjLCF3pZzGkl/kOXj7pFSAsy4R1 uGZLKFqnXYl04WAi
Ll+seJ4+mu6AsBDWwGpQO+1KFYFB9RaGVdDDMjhUhHX0clC0zGKoplfCldO1 YwGEhbDM1VXWbPcq
qZX6P+5ZBfSsgJIQYRmUXjWF5Wdnak+mijJMgLAsFtZhmi1tK3H8NLsKNl2C Q9MdEBbCGqSrCuni
5mDjaVc03QkVYUF/NdgQlvpsMGCFMxZAWAjLEF11nqbPF3nzCmEBJSHCMjW9 SrIqddorZAUI6wjC
sqzZ4g1Or5KGVaKr7WtBmu6AsBBWU1hhRymYnlR0j94VTXdCRViu4zWE1ZJe JQ0rPzsHMt12LICw
ENb2wqotugk7u1dpdrXuImcASkIYlGF1zGVQZ5EpJl7hK0BYxxCWNUn28PQq qQaz/VH3MRZNd0BY
rh9Z5Qyre0fnwN95Mxya7oSKsCgJh6RX2YaDnAAZCyAshGWCsHp0te6GOACU hNCVUFXpSa+EyE7R
AICwDiYsg5NsTxtrXzWY+WpfY9F0B4SFsNqrwWS1YJCesG/3yVc03QkVYblU C2rPzN6SXqUT29P9
UVN30W/HAggLYW2YWmkWOrekVkkFGOSyYocJoCREWDsLq5lepZt3ifQkfdl+ E4wgIKyDCsu4JLuo
BesLnZuxZpNDk5nt+WeDPsMKCAthbZxhNdIrjbD89BzIftZsN+TMojTdCRVh uVEP6rvtzQdmmznn
G3iV+1eABRAWwtopw9J+OOjn289n6RUAJSHC2lxWI9OrdJIowgKEdXhhmZZk N4VVpFen1vTKOFvR
dAeEdfgjy8tbWJ4uvToV2VXSYPfTqaIMqyOx0sOiJDRKWI2Ou657VWRXAaeQ wQIIC2HtmWFVSsKm
rormlR8IWu1ASYiwdk2vSsIivQKEhbAMTbL7T9peWYljdnpF0x0Q1uGPrJ49 cfLsKlk4aHJ6RdOd
UBGWKyWhaD2PTLYSh+YVFkBYCGvv1KrnpO1+Ob1CWEBJiLD2E1aRYbWeBTmb 2U56BQjLTWGZkWRX
MqzWk7af4rOK7rrd4BFrF0BYCGuUrSrzGTo2mUjmMlhRD9J0J1SEdeySUF11 7onjq9PIUA5iAYSF
sPYzVenjwd4NnREWUBIirP1TKyF6thxkQ2dAWAhr5yS7LKweXaXTGuyYz0DT HRDW0Y6sSre9O71K
YrUnwaLpTqgI63D1YGlue296JfJ1hAwcFkBYCGuXerBzJU5ZV9mGXqzJAUpC hLVretVTDeYTr/z0
PwCE5aiwdkuySytxWh7hq/SqtJnzyZ41hDTdAWEd7cjqSq8SNwWifBoZv3TJ sLoYKz0sSsKdM6z2
dc7pTFEhKAOxAMJCWCYIq2vhYLZuUHAyGaAkRFgGOCtsqQVVNej7+fbzGAsQ FsLaNcmO86vKjl5l
YxVnvfKpXQBhIaz9j6ww9LSlYPxfkJ5ctGoshOV6rPSwKAl381V9z9TCWEHg ZxtOME5YAGEhLGN8
pUuyAjpXAAjLENLTirYJK+5eISoAhGVGku3lc9u9lmqwzVjULoCwBj0vYtgV wuoXVj75yhOlus9P
qsGuaVcIy/VY6WEN9JX6GnBFSdgjq8pkBk8UxvKr1SA1IRYgVIS1u7B0kxmy IjDAVABLlIQik5Ls
uSqVkAmnWLQmX4hN3y2uBr3GHZGtTn4QxD/66owMxo/aoS4EY2DYReGP6bqS otdWVmZYGybZSTVY
+mAwr/7iqaKV4pCCAGBO031gZYiwOojnMoiKrdLPA5MPB0Xv5CuERWOIHtZg X9HDmtW8qk8VTaey
++nCQZpXWIBQFxOWzGrCw01r2MhWXlgrB0s7SgScPwZgyZLQ1PexNr1KhKVI q0GMBYCwjEiyi/Sq
KazANytW92oXQFgIq2SrRFeisRInSa/GpFYIy/VY6WFREq5cC4piLoPX9BXn j8EChIqwTLFVugO9
p1nnLPzipO0AgLBMEFbXaWTYdx4AYZmQZKsKMOleCV16JdQeE2OFRe0CCAth rSGsZBOv2lkZyulV
elZRhEWshIqwdq8Fi/RKJ6xs4SAFIRYgVIS1t7BSXelrQZVeJb5itigAwto5 u0pW4rTZKkuvOFEf
AMLaNcn28uZVKHrSKwoCAIRlhrBCr2WL1LTZjrCIlVAR1v71YJFe1U0lsrP0 UQNiAUJFWKbYSk0V
1eVW+RZeAICwjHBWvhKnaSw/38ILABDW3kl2PpdBizovw9yZV9QugLAQ1mKE WlXFF0F+VlHfDQvQ
dCdUhGVyLdiy46BIT8sg2CIVCxAqwjJHWNrmVbKfs+9jKgCEZY6wwlB/z8hz igIAwlo1c02nilbP
eZWVf+VikIIAAGEZcGSVysHsrDHphoN+uv8gwiJWQkVYRlA5MYMSlTo3Xzyx 3U83TKUqxAKEirDM
8FX1pO2prErrBpEVAMIyx1dVVH41c5kzACCs5TPX4tPBPJuK9+9aZ/t5ahdA WAhrofTKz3ylmu1r
dK4Qluux0sOiJFwivUpcpf5TvvJ9OldYgFARllm+qmdXyX9MFQWgJDQ2vUqN 5acTRAN+zQAQllmZ
a2MpTmqsIDAvVmoXQFhuCyvU2io9iwwWoOlOqPSwTK0GE2OphYP0rmwEYSGs Y/tKCF09GHAOZABK
QuPTq8RZNNsBEJZpmWvLea9W7l5RuwDCQlijj6y28/RtUw0iLIbVrVAR1hrV YLb9PP0rO0FYCOuY
vurUFQBQEpqfXvFrBYCwDMtcTUivqF0AYSEsa9IrhMWwuhUqwlq2GqQcPAII C2G5oCu6VwCUhMb5
SpBeASAsKzJXo6pBahdAWAjLmmoQYTGsboWKsKgGAWEhrCMKi2Y7ACUh6RUA IKxlM1cT0ytqF0BY
CGtEObhveoWwGFa3QkVYs8pBqsFDgrAQ1jF9xbENQElomq5C0isAhGVH5mpw ekXtAggLYVmTXiEs
htWtUBFWXzlINegeCAthWaqrEF0BUBJarCs+GwRAWOZlrhZUg9QugLAQljXV IMJiWN0KFWG1lYNU
g+6CsBCWZbqi2Q5ASUh6BQAIa8nM1aL0itoFEJbbwrKqGkRYDKtboSIsqkFA WAjLTmHRbAegJLRF
WKRXAAjLEmGl6dXJovSK2gUQlqPCCrVHVoAF3BMWw2psqAirml4Ji9IrwAII y01hoSsASkJbhEWz
HQBhWSKsWnp1sii9onYBhOWYsELtkRVgAXeFxbAaG6rzwqJ7BQgLYdkiLLpX AJSElgiL9AoAYVki
rAPoitoFEJYjwmqrBukOOy8shtXYUJ0VFtUgYAGEZYuwaLYDUBJaIizSKwCE ZYuwutMrmi2uhwoI
yyRh9aRXCMt5YTGsxoY6WSRSJpfquuvKNGFpy0GqQUBYRxZW5iT11XVlmrDC rmoQAI5YEiY+slBY
pFcA7glL5MKSiZtar4qMLOUUZ4Y7XYSh5o5gx4DmXpzsCfdk0cAKe38jDnpR +GOmsGSftEzKsIal
V3SHXQ+VYT1eD8vCklDXbacaBCyAsEwUFroCcLWHlZWT1kxr0KRX6ArAEWEZ +z4L6Ipmi+uhAsLa
VVi6avAYRxbCYljdCtUBYVENAhZAWLYIa1x6BQCUhLsJi/QKAGHZIqwpuqLZ 4nqogLB2EZYuvTrW
kYWwGFa3Qj2wsKgGAQsgLFuERbMdgJLQEmGRXgEgLFuENSe9otnieqiAsDYV 1rz0CmE5LyyG1dhQ
jygsqkHAAgjLEmFNm8sAAJSEm78PzXYAhGXL+ywxl4Fmi+uhAsLa5H2WSa8Q lvPCYliNDfVIwmKq
KGABhGWJsOheAVAS2vI+6AoAYdnyPgvOZaDZ4nqogLDWfZ9mOTgjvUJYzguL YTU21EMIi6migAUQ
li3CWjK9AgBKwhXfR1MO8tcKgLCMfJ/l0yuaLa6HCghrpfdppFcLVIMIy3lh MazGhmq3sKgGAQsg
LEuEtUZ6BQCUhCu8D812AIRly/usN5eBZovroQLCWvh91kyvEJbzwmJYjQ3V TmExVRSwAMKyRFg0
2wEoCS15H5rtAAjLkvcJ168Gaba4HiogrIXeZ4vzMiAs54XFsBobqlXCCmm2 AxZAWHYIix1SASgJ
bXkfTtoOgLAseZ8Nq0GaLc4PKyCsxdMrjiyERaz0sEwUFs12wAKEaomwaLYD gC3CotkOAJYIa4dq
kGaL88MKCGuh9GqD7AphMayEamqoRguLdc6ABQjVFmHx2SAAWCIsznoFAJYI a8dqkGaL88MKCMue
ahBhMayEamqoRgqLZjtgAUK1RVg02wHAEmHRbAcAW4S1fzVIs8X5YQWENdFX O6RXCIthJVRTQzVL
WM1ykH9SAAsQqpnConsFALYIi/QKACwRVr0c3C29otni/LACwhqZXu1YDSIs hpVQTQ3VEGE10iv+
KQEsQKiGCsuc9AoAKAk734f0CgAsEVZoWnpFs8X5YQWENbAaNCC9QlgMK6Ga GurOwgrpXgEWIFRL
hIWuAMASYdFsBwBbhGVqekWzxflhBYRlT3qFsBhWQjU11J2ERbMdsACh2iIs dAUAlgirll6hKwAw
Vlim64pmi/PDCghLn16ZmF0hLIaVUE0NdWth1dIr/sUALECopgrL/PQKACgJ 1fuEpFcAYImwbPls
kGaL88MKCMueahBhMayEamqo+0wcpRoELEColgjL8Ga7tOjIkoTKsDoV6g7C CvgV4NeVWAnVDmGZ
P5eBI4thJVSEZUd6xZFFqAwrwkrfx4qpohxZDCuhIixrZrZzZDGshOqgsGRE +X0CfgX4dSVWQjX0
zxvbqjCWtGfhIEcWw0qozguLXwF+XYmVUI0WVqkmlAAAs1mzhSWkTf+sAoC7 VEtCAACEBQCwTE2I
rwAAAAAAAAAAAAAAAAAcxpLPDLPZtOaHm8SXxml2uHmI5o9sdTyNjrUYTuOH 1aJRLf/O2mCsNEbz
w83+7tM1UCaHW4Rq/MhWx9PoWEsxmj6sFo2qhcKSWahGhyuFNcKSpQwLYS0f qiW/sPYJS9qSYSWD
anq4mbCk+eHW3CpNz10tGdaaBaTRObYU1oxqXhbYMvG9GFgLhCUtCDcPzYKR zRpCVgyrtOcXtmor
SsJ1/t2yIcOyINyqsCwI1ZLWoE2/sPSwtmhiICzHQq22sI3vYVkUqm3CsmYp tCUfvdY+zZaGNzAs
CZVh3eKg4qwIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADHRrKQDADsEZY9 pxkCAIRVOrGvfiMF
1vYDgEklYfPUSI3zJdl19iQAOGiGJTRb6FQTrdJjJF0vADBKWIW0hN5nAACG CEt7tl9KQgAwSlg0
3QEAAAAAAABgBv8f8Geb3ONPcpcAAAAASUVORK5CYII=

--Boundary-00=_Z3GdJJwjPdG45Jq--

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Torsten Foertsch schrieb:
> Of course it depends on your setup. If you configure only a small
> number of interpreters then the overall memory footprint can be lower
> then with prefork where each process runs its own interpreter. But the
> effect of the copied interpreters outweighs this very soon.

I see. Guess I have some reading to do to better understand what
"interpreters" as in Perl ithreads really means.

> Measuring memory consumption on Linux is not simple. You can't simply
> add up all RSS or VSZ figures or something like this. I know of 2
> approaches that work more or less well.
>
> 1) call mlockall in the parent and then use /proc/PID/smaps (or
> Linux::Smaps) to measure shared memory
>
> 2) take a system with enough RAM and nothing else to do, start apache
> with different numbers of children and watch the free memory with
> vmstat while doing so. Then you can estimate the amount of memory that
> is consumed by one apache process. See the attached picture for
> example.

Thanks for sharing this.

> You can run recovery in a separate process spawned at startup by the
> parent.

That's a good idea. And a simple one at that. Thank you!

Michael

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Mark Hedges schrieb:
>
> Just curious, did you try eliminating every other module
> that you might be loading (anywhere in any module loaded by
> Apache)? i.e. did you try a simple setup that only tries
> to open your database and does absolutely nothing else and
> loads no other CPAN modules. With "use strict" and "use
> warnings FATAL => 'all'" of course. --mark--

Time::HiRes was loaded in addition to the Apache2:: modules.
FATAL warnings weren't on, just plain warnings.

I changed the setup as follows:

use strict;
use warnings FATAL => 'all';

use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Const -compile => qw( OK );
use Apache2::RequestUtil ();
use Apache2::Util ();

use Apache2::Request ();

use Sleepycat::Db;
use Sleepycat::DbXml;

I dropped the query I was doing, which took about 0.7 seconds. So now
it's just opening the environment. Still seeing SEGVs.

I think the failure is related to how threads or processes are torn
down. What I'm getting is the following:

Database handles still open at environment close
Open database handle: tv.dbxml/structural_stats
Open database handle: tv.dbxml/secondary_document_statistics_decimal
Open database handle: tv.dbxml/secondary_document_index_decimal
Open database handle: tv.dbxml/secondary_document_statistics_date
Open database handle: tv.dbxml/secondary_document_index_date
Open database handle: tv.dbxml/secondary_document_statistics_string
Open database handle: tv.dbxml/secondary_document_index_string
Open database handle: tv.dbxml/secondary_document
Open database handle: tv.dbxml/content_document
Open database handle: tv.dbxml/secondary_dictionary
Open database handle: tv.dbxml/primary_dictionary
Open database handle: tv.dbxml/secondary_sequence
Open database handle: tv.dbxml/secondary_configuration
[Tue Jan 20 17:47:39 2009] [notice]
child pid 17142 exit signal Segmentation fault (11)

What does this mean? There is a file called tv.dbxml, which is the
Berkeley XML database (XmlContainer). The structural_stats etc are
named databases within that database files, as *file and *database
in open:

int
Db::open(DbTxn *txnid, const char *file,
const char *database, DBTYPE type, u_int32_t flags, int mode);

http://www.oracle.com/technology/documentation/berkeley-db/d b/api_cxx/frame.html

I'm just realizing Google doesn't know about these database names yet.

The error message is pretty explicit about what's wrong here: "Database
handles still open at environment close". Indeed, I do not explicitly
close the handles. In a batch script, this doesn't pose any problem;
mod_perl, however, isn't exactly equivalent to a batch script.

The problem is: Where in threaded mod_perl do I get a chance to do fixup
work, like closing handles?

I tried PerlChildInitHandler/PerlChildExitHandler, only to notice
that I then run into the issues of nothing being global, unless using
threads::shared, which caused even more weird issues (remember the
PerlChildInitHandler seemingly *blocking* on $DbEnv->open).

So I went back to Mark's and Craig's initial recommendation of using
package level my-variables (which are local to the thread) to hold the
handles, initialize them lazily, and then using them as long as needed.

The issue *seems* to be (I could be wrong) that in order for Berkeley to
be happy, I'd have to be able to have some fixup code run the moment a
thread decides it has to leave the stage, so the handles stored in the
package-level my-variables can be closed in a controlled fashion.

Can I have code run at that instant?

Or any other ideas what's going on?

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

A sidenote:

I've built more or less the same Apache as a prefork MPM, and I'm not
seeing any SEGVs any more.

On the other hand, performance isn't what you'd call great, as access to
the database is effectively serialized. Thirty queries will take thirty
seconds, regardless of concurrency level. This may have been a bit
faster in the worker MPM. (But only due to the SEGVs, unfortunately,
which came across pretty good, performance-wise.)

I think a single-process, multi-threaded approach (as in Java) will work
best for Berkeley DB.

Michael Ludwig

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

On Tue, Jan 20, 2009 at 4:19 PM, Michael Ludwig wrote:
> On the other hand, performance isn't what you'd call great, as access to
> the database is effectively serialized.

Are you sure about that? Berkeley DB doesn't work that way. It has
read and write locks which can be set to work at a (memory) page level
rather than a database level.

> I think a single-process, multi-threaded approach (as in Java) will work
> best for Berkeley DB.

I don't know about dbXML, but Berkeley DB is designed for
multi-process use. It uses shared memory and locks to coordinate.

- Perrin

Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl

Perrin Harkins schrieb:
> On Tue, Jan 20, 2009 at 4:19 PM, Michael Ludwig
> wrote:
>> On the other hand, performance isn't what you'd call great, as access
>> to the database is effectively serialized.
>
> Are you sure about that?

I'm sure about the queries seeming *effectively* serialized, judging
from the timing using the ab tool to test with different concurrency
levels.

> Berkeley DB doesn't work that way. It has read and write locks which
> can be set to work at a (memory) page level rather than a database
> level.

I know. I don't think this is a limitation of Berkeley DB or DB XML at
all. I think it may stem from me being unable to set up the environment
properly in the mod_perl context.

The query in question uses an index and runs in a transaction flagged
DB_READ_COMMITTED. There aren't any concurrent writes, though.

>> I think a single-process, multi-threaded approach (as in Java) will
>> work best for Berkeley DB.
>
> I don't know about dbXML, but Berkeley DB is designed for
> multi-process use. It uses shared memory and locks to coordinate.

Berkeley DB XML is built on top of Berkeley DB, so it is the same in
this regard, as can be seen on slides 12 through 17 (especially 14) of
this nice, new video tutorial:

http://www.oracle.com/technology/products/berkeley-db/tutori al-berkeleydb-ds.html

Michael Ludwig