How to pre-open files in apache"s processes

Hello,

I want to pre-open files in apache's processes.
So every modperl handlers will use the file handle directly, instead
of re-openning the files each time when the requesting is coming.

Currently I do it with:

package Fileopen;
use strict;
require Exporter;

our @ISA = qw(Exporter);
our @EXPORT = qw($filehandle);
our $filehandle;

open $filehandle,"<","/path/to/big.file" or die $!; # open for
reading only, the file itself is readable only.


1;


Then in startup.pl:

use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Request ();
use Fileopen;
# use others...

1;


Last in each handler:

package Myhandler;

use strict;
use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Const -compile => qw(OK);
use Apache2::Request ();
use Fileopen;

sub handler {
my $r = shift;
our $filehandle; # exported from Fileopen
# deal with $filehandle
...
}

1;


I'm not sure, am I going on the correct way for the purpose?
Thanks for the helps.

Regards.

Re: How to pre-open files in apache"s processes

Can you confirm the error you are encountering using your current method ?

I would like to understand what you are trying to achieve by having a global file handle - do you want to have modular code or does the content of the file somehow determine which handler to be used ?

You can have modular code by keeping the open in a init method in your package, push the package in @ISA and call SUPER::init from all the handler methods.


I hope you do realize the actual opening of the file doesn't take any appreciable amount of resources unless it is over a NFS over WAN.




- Mithun


----- Original Message -----
From:Xiao Lan
To:modperl@perl.apache.org
Cc:
Sent:Wednesday, December 1, 2010 7:58:32 AM
Subject:How to pre-open files in apache's processes

Hello,

I want to pre-open files in apache's processes.
So every modperl handlers will use the file handle directly, instead
of re-openning the files each time when the requesting is coming.

Currently I do it with:

package Fileopen;
use strict;
require Exporter;

our @ISA = qw(Exporter);
our @EXPORT = qw($filehandle);
our $filehandle;

open $filehandle,"<","/path/to/big.file" or die $!; # open for
reading only, the file itself is readable only.


1;


Then in startup.pl:

use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Request ();
use Fileopen;
# use others...

1;


Last in each handler:

package Myhandler;

use strict;
use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Const -compile => qw(OK);
use Apache2::Request ();
use Fileopen;

sub handler {
my $r = shift;
our $filehandle; # exported from Fileopen
# deal with $filehandle
...
}

1;


I'm not sure, am I going on the correct way for the purpose?
Thanks for the helps.

Regards.

Re: How to pre-open files in apache"s processes

On Thu, Dec 2, 2010 at 3:38 AM, Mithun Bhattacharya wrote:
> Can you confirm the error you are encountering using your current method ?
>
> I would like to understand what you are trying to achieve by having a global file handle - do you want to have modular code or does the content of the file somehow determine which handler to be used ?
>

Hi,

Thanks for the replying.
I was giving the example by openning a file.
In acutal I was using this module:

http://search.cpan.org/~sunnavy/IP-QQWry-0.0.16/lib/IP/QQWry .pm

For each client request if I open the data file:

my $qqwry = IP::QQWry->new('QQWry.Dat');

That will be slow IMO.

So I was thinking to pre-open the data file in each process of apache
and use it directly in each handler.

How about this case? Thanks.

Regards.

Re: How to pre-open files in apache"s processes

Am Do, 2.12.2010, 02:24, schrieb Xiao Lan:
> On Thu, Dec 2, 2010 at 3:38 AM, Mithun Bhattacharya
> wrote:
>> Can you confirm the error you are encountering using your current method
>> ?
>>
>> I would like to understand what you are trying to achieve by having a
>> global file handle - do you want to have modular code or does the
>> content of the file somehow determine which handler to be used ?
>>
>
> Hi,
>
> Thanks for the replying.
> I was giving the example by openning a file.
> In acutal I was using this module:
>
> http://search.cpan.org/~sunnavy/IP-QQWry-0.0.16/lib/IP/QQWry .pm
>
> For each client request if I open the data file:
>
> my $qqwry = IP::QQWry->new('QQWry.Dat');
>
> That will be slow IMO.
>
> So I was thinking to pre-open the data file in each process of apache
> and use it directly in each handler.
>
> How about this case? Thanks.
>
> Regards.
>

Hi.

You could do the following (and avoid any possible conflicts when using
the file descriptor globally - one thing that comes into my mind would be
seek):

In the perl file that holds your handler declare a variable on module level:

my $qqwry = undef;

In the handler instantiate the object if $qqwry is undefined:

if (not $qqwry)
{
$qqwry = IP::QQWry->new('QQWry.Dat');
}

This way the file should be opened only once in each apache process (which
should be good enough for performance reasons). The my $qqwry on module
level should be persistent in each process.

Hendrik

Re: How to pre-open files in apache"s processes

On Thursday, December 02, 2010 02:24:32 Xiao Lan wrote:
> I was giving the example by openning a file.
> In acutal I was using this module:
>=20
> http://search.cpan.org/~sunnavy/IP-QQWry-0.0.16/lib/IP/QQWry .pm
>=20
> For each client request if I open the data file:
>=20
> my $qqwry =3D IP::QQWry->new('QQWry.Dat');
>=20
> That will be slow IMO.
>=20
> So I was thinking to pre-open the data file in each process of apache
> and use it directly in each handler.

I don't think the module is slow because of opening a file, especially on=20
local disk. Apache does that all the time.

On the other hand there should be no problem with opening a file and keepin=
g=20
it open over the lifetime of the apache child. Apache itself does that. It=
=20
opens its logfiles only once. Also, modperl applications often cache databa=
se=20
connections. That means the database socket is opened only once for multipl=
e=20
requests. The only thing to think about is what happens if the database is=
=20
updated. If the original file is removed and a new one is written you'll st=
ill=20
keep the old one. If the original file is written to any index that you kee=
p=20
in RAM may be invalid.

But, looking at the module I think it can be slow because it relies heavily=
on=20
seek(). A better approach would be to mmap() the file. In perl it would loo=
k=20
then like a big string. My favourite tool for that is File::Map.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: How to pre-open files in apache"s processes

On Thu, Dec 2, 2010 at 5:58 PM, Hendrik Schumacher wrot=
e:

>
> In the perl file that holds your handler declare a variable on module lev=
el:
>
> my $qqwry =3D undef;
>
> In the handler instantiate the object if $qqwry is undefined:
>
> if (not $qqwry)
> {
>  $qqwry =3D IP::QQWry->new('QQWry.Dat');
> }
>
> This way the file should be opened only once in each apache process (whic=
h
> should be good enough for performance reasons). The my $qqwry on module
> level should be persistent in each process.
>


Hi,

I'm not sure, what's "the perl file that holds your handler"?
My hanlder is a perl package, which file will hold this package?

Thanks.

Re: How to pre-open files in apache"s processes

2010/12/2 Torsten Förtsch :

>
> But, looking at the module I think it can be slow because it relies heavi=
ly on
> seek(). A better approach would be to mmap() the file. In perl it would l=
ook
> then like a big string. My favourite tool for that is File::Map.
>

OK thanks for this info. I will take a look at File::Map.

Regards.

Re: How to pre-open files in apache"s processes

Am Do, 2.12.2010, 14:24, schrieb Xiao Lan:
> On Thu, Dec 2, 2010 at 5:58 PM, Hendrik Schumacher
> wrote:
>
>>
>> In the perl file that holds your handler declare a variable on module
>> level:
>>
>> my $qqwry = undef;
>>
>> In the handler instantiate the object if $qqwry is undefined:
>>
>> if (not $qqwry)
>> {
>> Â
$qqwry = IP::QQWry->new('QQWry.Dat');
>> }
>>
>> This way the file should be opened only once in each apache process
>> (which
>> should be good enough for performance reasons). The my $qqwry on module
>> level should be persistent in each process.
>>
>
>
> Hi,
>
> I'm not sure, what's "the perl file that holds your handler"?
> My hanlder is a perl package, which file will hold this package?
>
> Thanks.
>

"In the perl file that holds your handler" means your perl package file.
Sounds more difficult as it was meant. The important thing is that the
variable has to be declared on package level, not inside a sub routine.

Re: How to pre-open files in apache"s processes

On Thu, Dec 2, 2010 at 9:39 PM, Hendrik Schumacher wrote:

>
> "In the perl file that holds your handler" means your perl package file.
> Sounds more difficult as it was meant. The important thing is that the
> variable has to be declared on package level, not inside a sub routine.
>
>

AH thanks a lot for this point.
This sounds simpler much than my original idea.
Nice, thanks.

regards.

Re: How to pre-open files in apache"s processes

On Thursday, December 02, 2010 14:26:36 Xiao Lan wrote:
> > But, looking at the module I think it can be slow because it relies
> > heavily on seek(). A better approach would be to mmap() the file. In
> > perl it would look then like a big string. My favourite tool for that is
> > File::Map.
>=20
> OK thanks for this info. I will take a look at File::Map.

If I were you, I'd really consider to convert the database into an MMapDB, =
see=20
http://search.cpan.org/perldoc?MMapDB

You can create the MMapDB object in a section in the httpd.conf or i=
n a=20
ChildInit handler or something similar:


use MMapDB;
# use a global variable here to make it accessible from everywhere
$My::DB=3DMMapDB->new(filename=3D>..., readonly=3D>1);


Then in your handler (simplest case):

use MMapDB qw/:mode/;
$My::MMapDB->start; # update the object if the database file has changed
$My::DB->datamode=3DDATAMODE_SIMPLE;
($value)=3D@{$My::DB->main_index->{$key} || []};

If the extra step of converting the QQWry.Dat into a QQWry.mmdb can be buil=
t=20
into your work flow that would be the route I'd take.

Otherwise I'd look for a C library to that database and build a perl interf=
ace=20
layer in XS.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: How to pre-open files in apache"s processes

2010/12/2 Torsten Förtsch :
> On Thursday, December 02, 2010 14:26:36 Xiao Lan wrote:
>> > But, looking at the module I think it can be slow because it relies
>> > heavily on seek(). A better approach would be to mmap() the file. In
>> > perl it would look then like a big string. My favourite tool for that =
is
>> > File::Map.
>>
>> OK thanks for this info. I will take a look at File::Map.
>
> If I were you, I'd really consider to convert the database into an MMapDB=
, see
> http://search.cpan.org/perldoc?MMapDB
>

Yup that's a good idea.
But I dont know how to convert the original database to MMapDB. :-)

Thanks.