WWW::Mechanize & caching

Hi all,

I hope I did not miss an obvious solution to the following:

I want a *caching* version of WWW::Mechanize.

- No, WWW::Mechanize::Cached does not cut it for me. It does not
honor any HTTP cache control headers. And, also, I think that
caching better belongs to the UserAgent. Here it is much more
reusable anyway.

- There is a LWP::UserAgent::WithCache, but that is also agnostic
about HTTP. Even if that is fixed, I see it difficult to use it
together with WWW::Mechanize, as both are proper subclasses
of LWP::UserAgent.

I could do a

use WWW::Mechanize;
@WWW::Mechanize::ISA = ('LWP::UserAgent::WithCache');

my $mech = WWW::Mechanize->new();

but this looks like an awful hack to me.

Is it possible to do something like this:

use LWP::UserAgent::WithCache;
my $ua = new LWP::UserAgent::WithCache (...specify caching...);
use WWW::Mechanize;
my $mech = new WWW::Mechanize (ua => $ua,
agent => 'I gather');

?

\rho

Re: WWW::Mechanize & caching

>
> I want a *caching* version of WWW::Mechanize.
>
> - No, WWW::Mechanize::Cached does not cut it for me. It does not
> honor any HTTP cache control headers. And, also, I think that
> caching better belongs to the UserAgent. Here it is much more
> reusable anyway.

Patches are certainly welcome. Anything you want to add to it, I'm
glad to take a look at. I don't want WWW::Mechanize::Cached to mess
with LWP::UserAgent::WithCache, though.


--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

Re: WWW::Mechanize & caching

On Sun, Feb 20, 2005 at 07:35:47PM -0600, Andy Lester wrote:
> >
> >I want a *caching* version of WWW::Mechanize.
> >
> > - No, WWW::Mechanize::Cached does not cut it for me. It does not
> > honor any HTTP cache control headers. And, also, I think that
> > caching better belongs to the UserAgent. Here it is much more
> > reusable anyway.
>
> Patches are certainly welcome. Anything you want to add to it, I'm
> glad to take a look at. I don't want WWW::Mechanize::Cached to mess
> with LWP::UserAgent::WithCache, though.

Andy,

OK, but then we have the problem that caching is implemented only
inside WWW::Mechanize::Cached and that it cannot be reused by people
who only want to use a caching version of LWP::UserAgent.

I would rather suggest not to invest into WWW::Mechanize::Cached and
prop up LWP::UserAgent::WithCache to use it as a proper replacement
for LWP::UserAgent:

use LWP::UserAgent::WithCache;
my $ua = new LWP::UserAgent::WithCache (...specify caching...);
use WWW::Mechanize;
my $mech = new WWW::Mechanize (ua => $ua,
agent => 'I gather');

WWW::Mechanize would then use a 'component' user agent (instead of
being a subclass). Agreed, this is major design change, but one which
would give us more options.

For you to consider.

\rho

PS: Another option we have is to use HTTP::Cache::Transparent which
silently hooks itself into the request fetching. But that also
does not yet support proxy cache headers (yet).

Re: WWW::Mechanize & caching

On Mon, Feb 21, 2005 at 08:27:38AM +1000, Robert Barta wrote:
> Hi all,
>
> I hope I did not miss an obvious solution to the following:
>
> I want a *caching* version of WWW::Mechanize.

Why don't you just use a caching proxy server? Squid?

--
Reinier Post

Re: WWW::Mechanize & caching

On Thu, Feb 24, 2005 at 11:07:00PM +0100, Reinier Post wrote:
> On Mon, Feb 21, 2005 at 08:27:38AM +1000, Robert Barta wrote:
> > Hi all,
> >
> > I hope I did not miss an obvious solution to the following:
> >
> > I want a *caching* version of WWW::Mechanize.
>
> Why don't you just use a caching proxy server? Squid?

First, we need a bit more control on the caching policy. Reconfiguring
a squid remotely is a bit brittle :-) But, more importantly, we cannot
assume that a proxy/cache is at every user site where the agent is
running.

\rho

Re: WWW::Mechanize & caching

On Fri, Feb 25, 2005 at 10:42:43AM +1000, Robert Barta wrote:
> On Thu, Feb 24, 2005 at 11:07:00PM +0100, Reinier Post wrote:
> > On Mon, Feb 21, 2005 at 08:27:38AM +1000, Robert Barta wrote:
> > > Hi all,
> > >
> > > I hope I did not miss an obvious solution to the following:
> > >
> > > I want a *caching* version of WWW::Mechanize.
> >
> > Why don't you just use a caching proxy server? Squid?
>
> First, we need a bit more control on the caching policy. Reconfiguring
> a squid remotely is a bit brittle :-) But, more importantly, we cannot
> assume that a proxy/cache is at every user site where the agent is
> running.

I was assuming you'd put it on the client side.
But perhaps Squid won't run there.

--
Reinier