HTTP::TransparentCache uploaded to CPAN

I have now uploaded my HTTP::TransparentCache module to CPAN. It
implements a persistent cache for HTTP get-requests. The module should
work seamlessly with all code that uses LWP to make HTTP requests. All
you have to do is add

use HTTP::TransparentCache;
HTTP::TransparentCache::init({ BasePath => '/tmp/cache*' } );

at the top of your perl-program. After that, all get-requests, through
LWP::Simple as well as the more advanced LWP-classes, will be cached.

This initial version does not cache any requests where the response is
saved in a file by LWP::UserAgent or where data is returned with a
callback. I intend to fix this in a later release, but for now these
requests are sent directly to LWP::UserAgent bypassing the cache.
Caching is also disabled for any requests containing a Range-header.

Please test the module with your favorite project. There is a simple
example in examples/check_server that can be used to test if your
http-server supports caching.

Note that the HTTP::TransparentCache module always checks with the
server before returning data from the cache. This means that you never
run the risk of receiving old data from the cache.

The module will hopefully show up on CPAN soon. In the meantime, you can
read the pod at http://www.holmlund.se/HTTP-TransparentCache.html and
download the code from

http://www.holmlund.se/HTTP-TransparentCache-0.1.tar.gz

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

What happens if you try to use it without calling init first? :)

Perhaps this situation might be avoided if BasePath could be specified
at module load time, like so:

use HTTP::TransparentCache ( BasePath => '/tmp/cache*' );

Of course, this precludes two pieces of code in the same process from
using different caches, but that restriction already exists with the
current init model. In order to go beyond that, you'd have to allow
cache configuration at the LWP::UserAgent object level, and I'm not
quite sure how to best implement that.

Just food for though. Otherwise, congrats on the first release! The
Transparency design is great.

-ofer

Mattias Holmlund wrote:

>I have now uploaded my HTTP::TransparentCache module to CPAN. It
>implements a persistent cache for HTTP get-requests. The module should
>work seamlessly with all code that uses LWP to make HTTP requests. All
>you have to do is add
>
>use HTTP::TransparentCache;
>HTTP::TransparentCache::init({ BasePath => '/tmp/cache*' } );
>
>at the top of your perl-program. After that, all get-requests, through
>LWP::Simple as well as the more advanced LWP-classes, will be cached.
>
>This initial version does not cache any requests where the response is
>saved in a file by LWP::UserAgent or where data is returned with a
>callback. I intend to fix this in a later release, but for now these
>requests are sent directly to LWP::UserAgent bypassing the cache.
>Caching is also disabled for any requests containing a Range-header.
>
>Please test the module with your favorite project. There is a simple
>example in examples/check_server that can be used to test if your
>http-server supports caching.
>
>Note that the HTTP::TransparentCache module always checks with the
>server before returning data from the cache. This means that you never
>run the risk of receiving old data from the cache.
>
>The module will hopefully show up on CPAN soon. In the meantime, you can
>read the pod at http://www.holmlund.se/HTTP-TransparentCache.html and
>download the code from
>
>http://www.holmlund.se/HTTP-TransparentCache-0.1.tar.gz
>
>/Mattias
>
>

Re: HTTP::TransparentCache uploaded to CPAN

On m=E5n, 2004-10-04 at 23:14, Ofer Nave wrote:
> What happens if you try to use it without calling init first? :)
>=20

Your request is sent out uncached, i.e. as if you didn't do "use
HTTP::TransparentCache" at all.

> Perhaps this situation might be avoided if BasePath could be specified=20
> at module load time, like so:
>=20
> use HTTP::TransparentCache ( BasePath =3D> '/tmp/cache*' );
>=20

Yes, that was suggested by Randal L. Schwartz as well. The reason I
didn't do that is that I think that you want to have the BasePath as a
user-configurable parameter. This parameter should probably be loaded
from a configuration file and then you have to defer the loading of
HTTP::TransparentCache until after you've loaded the configuration file.
Calling init() explicitly seemed to involve less "magic" and would work
more like a normal perl module.

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

Mattias Holmlund declared:

> On mån, 2004-10-04 at 23:14, Ofer Nave wrote:
>
> Yes, that was suggested by Randal L. Schwartz as well. The reason I
> didn't do that is that I think that you want to have the BasePath as a
> user-configurable parameter. This parameter should probably be loaded
> from a configuration file and then you have to defer the loading of
> HTTP::TransparentCache until after you've loaded the configuration
> file. Calling init() explicitly seemed to involve less "magic" and
> would work more like a normal perl module.
>
> /Mattias

I think you could do it to work both ways. You could make the module to
allow specifying the temp path in the "use" line, and also let it allow
specifying it using an init method.

Some people might prefer a unique temp path for all their code and might
like one way, while others might prefer specifying it separately for each
project or perl program, and they might like the other one.

Teddy

Re: HTTP::TransparentCache uploaded to CPAN

>>>>> "Octavian" == Octavian Rasnita writes:

Octavian> I think you could do it to work both ways. You could make
Octavian> the module to allow specifying the temp path in the "use"
Octavian> line, and also let it allow specifying it using an init
Octavian> method.

Yes, it would be much more convenient to be initialized on "use",
if you gave it the right parameters, or initialized later if not.

In fact, I'd call the method "configure", rather than "init", implying
that I could readjust it later.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: HTTP::TransparentCache uploaded to CPAN

On tis, 2004-10-05 at 16:40, Randal L. Schwartz wrote:
> Yes, it would be much more convenient to be initialized on "use",
> if you gave it the right parameters, or initialized later if not.

Sorry for being stubborn, but I disagree with this. I'm not arguing
against you to avoid work; we're only talking about a handful lines of
code. To me, specifying the initialization parameters in the "use"-line
has two major drawbacks:

It is non-standard. The great majority of modules on CPAN uses the
"use"-line to specify which symbols to import. To use it for something
else would probably confuse a lot of users. I didn't even think it was
possible before we started this discussion.

It makes it impossible to export any symbols (conditionally) from
HTTP::TransparentCache later.

The only advantage I can see with doing initialization via the use-line
is to avoid a single line of code for each program that uses
HTTP::TransparentCache.

To me, these two drawbacks outweigh the convenience of getting rid of
the init-call by far. Unless you have a convincing argument for why
initialization via the use-line is superior, I will not implement this
feature. Sorry.

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

>>>>> "Mattias" == Mattias Holmlund writes:

Mattias> The only advantage I can see with doing initialization via the use-line
Mattias> is to avoid a single line of code for each program that uses
Mattias> HTTP::TransparentCache.

I'd like to be able to go from:

perl myprogram

to

perl -MHTTP::TransparentCache myprogram

and see the instant difference. If I have to add more code,
it'll be far less fun. I can even set

PERL5OPT="-MHTTP::TransparentCache"

and get all my programs sped up.

That's why.

If you fail to do that in your code, I shall have to release
a module called

HTTP::ReallyTransparentCache

that loads your code and calls your stupid initialization routine.
Darn it. Don't make me do that. Wasting my time for no reason. :(

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: HTTP::TransparentCache uploaded to CPAN

On lör, 2004-10-09 at 14:53, Randal L. Schwartz wrote:
> >>>>> "Mattias" == Mattias Holmlund writes:
> Mattias> The only advantage I can see with doing initialization via the u=
se-line
> Mattias> is to avoid a single line of code for each program that uses
> Mattias> HTTP::TransparentCache.
>=20
> I'd like to be able to go from:
>=20
> perl myprogram
>=20
> to
>=20
> perl -MHTTP::TransparentCache myprogram
>=20
> and see the instant difference. If I have to add more code,
> it'll be far less fun. I can even set
>=20

So THAT'S what you wanted to do. I didn't realize that, we only ever
talked about the "use HTTP::TransparentCache qw/.../" scenario.

That makes sense. I'll support that kind of syntax in the next version.
But I think it will have to be=20

perl '-MHTTP::TransparentCache qw(BasePath /tmp/cache)' yourscript.pl

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

On Saturday 09 Oct 2004 1:53 pm, Randal L. Schwartz wrote:

> I'd like to be able to go from:
>
> perl myprogram
>
> to
>
> perl -MHTTP::TransparentCache myprogram

Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(
Do any major distro's ship with this version yet?

-Nigel

--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
njh@despammed.com http://www.bandsman.co.uk

Re: HTTP::TransparentCache uploaded to CPAN

On sön, 2004-10-10 at 23:22, Nigel Horne wrote:
> Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(
> Do any major distro's ship with this version yet?

Debian testing does. But the requirement for perl 5.8.4 is probably
bogus. I'm running 5.8.4 and h2xs automatically inserts a requirement
for the perl it is running under. Try to change the use 5.008004 line at
the top of Makefile.PL and test if it works with your perl. If it does,
report back to me which version of perl you're running and I'll lower
the requirement in the next version.

/Mattias

Re: [despammed] Re: HTTP::TransparentCache uploaded to CPAN

On Monday 11 Oct 2004 04:56, Mattias Holmlund wrote:
> On sön, 2004-10-10 at 23:22, Nigel Horne wrote:
> > Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(

> >Try to change the use 5.008004 line at
> the top of Makefile.PL and test if it works with your perl.

That's not possible is it from -eshell is it? I am using FC2 (5.8.3) and YDL3.0.1 (5.8.0).

Look at this screen shot, I can't see anywhere I'm given the opportunity to change using this route:

[njh@njh njh]$ su -
Password:
[root@njh root]# !perl
perl -MCPAN -eshell

cpan shell -- CPAN exploration and modules installation (v1.7601)
ReadLine support enabled

cpan> install HTTP::TransparentCache
CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
Database was generated on Sun, 26 Sep 2004 14:05:33 GMT
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/01mailrc.txt.gz
Going to read /root/.cpan/sources/authors/01mailrc.txt.gz
Fetching with LWP:
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/modules/02packages.details.txt.gz
Going to read /root/.cpan/sources/modules/02packages.details.txt.gz
Database was generated on Sun, 10 Oct 2004 14:05:41 GMT
Fetching with LWP:
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/modules/03modlist.data.gz
Going to read /root/.cpan/sources/modules/03modlist.data.gz
Going to write /root/.cpan/Metadata
Running install for module HTTP::TransparentCache
Running make for M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar.gz
Fetching with LWP:
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/id/M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar .gz
CPAN: Digest::MD5 loaded ok
Fetching with LWP:
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/id/M/MA/MATTIASH/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/M/MA/MATTIASH/HTTP-Transparen tCache-0.1.tar.gz ok
Scanning cache /root/.cpan/build for sizes
Deleting from cache: /root/.cpan/build/Tk-804.027 (141.6>10.0 MB)
HTTP-TransparentCache-0.1/
HTTP-TransparentCache-0.1/examples/
HTTP-TransparentCache-0.1/examples/check_server
HTTP-TransparentCache-0.1/lib/
HTTP-TransparentCache-0.1/lib/HTTP/
HTTP-TransparentCache-0.1/lib/HTTP/TransparentCache.pm
HTTP-TransparentCache-0.1/MANIFEST
HTTP-TransparentCache-0.1/META.yml
HTTP-TransparentCache-0.1/Changes
HTTP-TransparentCache-0.1/t/
HTTP-TransparentCache-0.1/t/HTTP-TransparentCache.t
HTTP-TransparentCache-0.1/README
HTTP-TransparentCache-0.1/Makefile.PL

CPAN.pm: Going to build M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar.gz

Perl v5.8.4 required--this is only v5.8.3, stopped at Makefile.PL line 1.
BEGIN failed--compilation aborted at Makefile.PL line 1.
Running make test
Make had some problems, maybe interrupted? Won't test
Running make install
Make had some problems, maybe interrupted? Won't install

> /Mattias

-Nigel

--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
njh@despammed.com http://www.bandsman.co.uk

Re: [despammed] Re: HTTP::TransparentCache uploaded to CPAN

So do it manually. I rarely use the CPAN shell these days.

> wget
ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/id/M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar .gz
> tar xzvf HTTP-TransparentCache-0.1.tar.gz
> cd HTTP-TransparentCache-0.1
> vi Makefile.PL <--- make your changes
> perl Makefile.PL
> make
> make install

-ofer

Nigel Horne wrote:

>On Monday 11 Oct 2004 04:56, Mattias Holmlund wrote:
>
>
>>On sön, 2004-10-10 at 23:22, Nigel Horne wrote:
>>
>>
>>>Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(
>>>
>>>
>
>
>
>>>Try to change the use 5.008004 line at
>>>
>>>
>>the top of Makefile.PL and test if it works with your perl.
>>
>>
>
>That's not possible is it from -eshell is it? I am using FC2 (5.8.3) and YDL3.0.1 (5.8.0).
>
>Look at this screen shot, I can't see anywhere I'm given the opportunity to change using this route:
>
>[njh@njh njh]$ su -
>Password:
>[root@njh root]# !perl
>perl -MCPAN -eshell
>
>cpan shell -- CPAN exploration and modules installation (v1.7601)
>ReadLine support enabled
>
>cpan> install HTTP::TransparentCache
>CPAN: Storable loaded ok
>Going to read /root/.cpan/Metadata
> Database was generated on Sun, 26 Sep 2004 14:05:33 GMT
>CPAN: LWP::UserAgent loaded ok
>Fetching with LWP:
> ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/01mailrc.txt.gz
>Going to read /root/.cpan/sources/authors/01mailrc.txt.gz
>Fetching with LWP:
> ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/modules/02packages.details.txt.gz
>Going to read /root/.cpan/sources/modules/02packages.details.txt.gz
> Database was generated on Sun, 10 Oct 2004 14:05:41 GMT
>Fetching with LWP:
> ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/modules/03modlist.data.gz
>Going to read /root/.cpan/sources/modules/03modlist.data.gz
>Going to write /root/.cpan/Metadata
>Running install for module HTTP::TransparentCache
>Running make for M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar.gz
>Fetching with LWP:
> ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/id/M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar .gz
>CPAN: Digest::MD5 loaded ok
>Fetching with LWP:
> ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl /CPAN/authors/id/M/MA/MATTIASH/CHECKSUMS
>Checksum for /root/.cpan/sources/authors/id/M/MA/MATTIASH/HTTP-Transparen tCache-0.1.tar.gz ok
>Scanning cache /root/.cpan/build for sizes
>Deleting from cache: /root/.cpan/build/Tk-804.027 (141.6>10.0 MB)
>HTTP-TransparentCache-0.1/
>HTTP-TransparentCache-0.1/examples/
>HTTP-TransparentCache-0.1/examples/check_server
>HTTP-TransparentCache-0.1/lib/
>HTTP-TransparentCache-0.1/lib/HTTP/
>HTTP-TransparentCache-0.1/lib/HTTP/TransparentCache.pm
>HTTP-TransparentCache-0.1/MANIFEST
>HTTP-TransparentCache-0.1/META.yml
>HTTP-TransparentCache-0.1/Changes
>HTTP-TransparentCache-0.1/t/
>HTTP-TransparentCache-0.1/t/HTTP-TransparentCache.t
>HTTP-TransparentCache-0.1/README
>HTTP-TransparentCache-0.1/Makefile.PL
>
> CPAN.pm: Going to build M/MA/MATTIASH/HTTP-TransparentCache-0.1.tar.gz
>
>Perl v5.8.4 required--this is only v5.8.3, stopped at Makefile.PL line 1.
>BEGIN failed--compilation aborted at Makefile.PL line 1.
>Running make test
> Make had some problems, maybe interrupted? Won't test
>Running make install
> Make had some problems, maybe interrupted? Won't install
>
>
>
>>/Mattias
>>
>>
>
>-Nigel
>
>
>

Re: HTTP::TransparentCache uploaded to CPAN

Mattias Holmlund wrote:

>On lör, 2004-10-09 at 14:53, Randal L. Schwartz wrote:
>
>
>>>>>>>"Mattias" == Mattias Holmlund writes:
>>>>>>>
>>>>>>>
>>Mattias> The only advantage I can see with doing initialization via the use-line
>>Mattias> is to avoid a single line of code for each program that uses
>>Mattias> HTTP::TransparentCache.
>>
>>I'd like to be able to go from:
>>
>> perl myprogram
>>
>>to
>>
>> perl -MHTTP::TransparentCache myprogram
>>
>>and see the instant difference. If I have to add more code,
>>it'll be far less fun. I can even set
>>
>>
>>
>
>So THAT'S what you wanted to do. I didn't realize that, we only ever
>talked about the "use HTTP::TransparentCache qw/.../" scenario.
>
>That makes sense. I'll support that kind of syntax in the next version.
>But I think it will have to be
>
>perl '-MHTTP::TransparentCache qw(BasePath /tmp/cache)' yourscript.pl
>
>/Mattias
>
>
>
>
Or you could supply a reasonable BasePath by default, perhaps using
File::Spec->tmpdir();

-ofer

Re: HTTP::TransparentCache uploaded to CPAN

On sön, 2004-10-10 at 23:22, Nigel Horne wrote:
> On Saturday 09 Oct 2004 1:53 pm, Randal L. Schwartz wrote:
> Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(

Version 0.2 of HTTP::TransparentCache only requires Perl 5.6.1. That's
the oldest version of Perl that I had available to test with.

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

On lör, 2004-10-09 at 14:53, Randal L. Schwartz wrote:
> I'd like to be able to go from:
>=20
> perl myprogram
>=20
> to
>=20
> perl -MHTTP::TransparentCache myprogram
>=20
> and see the instant difference. If I have to add more code,
> it'll be far less fun. I can even set
>=20
> PERL5OPT=3D"-MHTTP::TransparentCache"
>=20
> and get all my programs sped up.

This is now supported in version 0.2 of HTTP::TransparentCache. See
perldoc for details.

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

On m=E5n, 2004-10-11 at 19:33, Ofer Nave wrote:
> Or you could supply a reasonable BasePath by default, perhaps using=20
> File::Spec->tmpdir();

Using tmpdir defeats the purpose of the module somewhat, since a tmpdir
is not guaranteed to be persistent between invocations Debian clears
/tmp on startup at least.=20

It is now possible to specify the BasePath via perl -M instead.

/Mattias

Re: HTTP::TransparentCache uploaded to CPAN

>>>>> "Nigel" == Nigel Horne writes:

Nigel> Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(

Uh why?

Nigel> Do any major distro's ship with this version yet?

People still wait for distros? :)

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: HTTP::TransparentCache uploaded to CPAN

>>>>> "Mattias" == Mattias Holmlund writes:

Mattias> Using tmpdir defeats the purpose of the module somewhat,
Mattias> since a tmpdir is not guaranteed to be persistent between
Mattias> invocations Debian clears /tmp on startup at least.

But it'd still be sped up for the current boot cycle. For example,
here's my uptime:

10:09PM up 91 days, 18:57, 1 user, load averages: 0.68, 0.65, 0.60

That'd be quite a lot of cache. :)

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: [despammed] Re: HTTP::TransparentCache uploaded to CPAN

On Monday 11 Oct 2004 19:59, Mattias Holmlund wrote:
> On sön, 2004-10-10 at 23:22, Nigel Horne wrote:
> > On Saturday 09 Oct 2004 1:53 pm, Randal L. Schwartz wrote:
> > Unfortunately HTTP::TransparentCache needs perl 5.8.4 :-(
>
> Version 0.2 of HTTP::TransparentCache only requires Perl 5.6.1. That's
> the oldest version of Perl that I had available to test with.

Well done, thanks. I'll give it a go.

> /Mattias

-Nigel

--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
njh@despammed.com http://www.bandsman.co.uk

unsunscribe

-----Original Message-----
From: Randal L. Schwartz [mailto:merlyn@stonehenge.com]
Sent: Tuesday, October 12, 2004 3:10 PM
To: libwww@perl.org
Subject: Re: HTTP::TransparentCache uploaded to CPAN

>>>>> "Mattias" == Mattias Holmlund writes:

Mattias> Using tmpdir defeats the purpose of the module somewhat,
Mattias> since a tmpdir is not guaranteed to be persistent between
Mattias> invocations Debian clears /tmp on startup at least.

But it'd still be sped up for the current boot cycle. For example,
here's my uptime:

10:09PM up 91 days, 18:57, 1 user, load averages: 0.68, 0.65, 0.60

That'd be quite a lot of cache. :)

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl
training!