problem w/ robotrules record parse

I've run into the same problem J and T ran into here:
http://www.mail-archive.com/libwww@perl.org/msg05452.html

Namely, malformed robots.txt files out there are being parsed somewhat =
draconianly by WWW::RobotRules.pm.

I've cobbled together a "fixed" version of RobotRules.pm -- how can I =
get it reviewed and ultimately blessed by the LWP community?

http://www.geocities.com/mvaneerde/RobotRules.pm.txt

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

Re: problem w/ robotrules record parse

On Wed, Sep 21, 2005 at 09:04:33AM -0700, Matthew.van.Eerde@hbinc.com (Matthew.van.Eerde@hbinc.com) wrote:
> I've cobbled together a "fixed" version of RobotRules.pm -- how can
> I get it reviewed and ultimately blessed by the LWP community?

It's not a community thing. Send a patch directly to Gisle, the owner
of LWP.

xoxo,
Andy

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

Re: problem w/ robotrules record parse

Andy Lester writes:

> On Wed, Sep 21, 2005 at 09:04:33AM -0700, Matthew.van.Eerde@hbinc.com (Matthew.van.Eerde@hbinc.com) wrote:
> > I've cobbled together a "fixed" version of RobotRules.pm -- how can
> > I get it reviewed and ultimately blessed by the LWP community?
>
> It's not a community thing. Send a patch directly to Gisle, the owner
> of LWP.

I still prefer patches to be posted to this list instead of me
directly. That way others might comment or pick up the patch for
their local use even if I'm not able to process it timely.

--Gisle

RE: problem w/ robotrules record parse

Gisle Aas wrote:
> Andy Lester writes:
>=20
>> Matthew.van.Eerde@hbinc.com (Matthew.van.Eerde@hbinc.com) wrote:=20
>>> I've cobbled together a "fixed" version of RobotRules.pm
>>=20
>> Send a patch directly to Gisle
>=20
> I still prefer patches to be posted to this list

Here's the patch, for the list.

http://www.geocities.com/mvaneerde/RobotRules.patch.txt

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

RE: problem w/ robotrules record parse

Matthew.van.Eerde wrote:
> Gisle Aas wrote:
>> Andy Lester writes:
>>=20
>>> Matthew.van.Eerde@hbinc.com (Matthew.van.Eerde@hbinc.com) wrote:
>>>> I've cobbled together a "fixed" version of RobotRules.pm
>>>=20
>>> Send a patch directly to Gisle
>>=20
>> I still prefer patches to be posted to this list
>=20
> Here's the patch, for the list.
>=20
> http://www.geocities.com/mvaneerde/RobotRules.patch.txt

And here's a smaller patch - only eleven new lines of code - which =
should have the same net effect.

http://www.geocities.com/mvaneerde/RobotRules.patch-3.txt

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

Re: problem w/ robotrules record parse

writes:

> And here's a smaller patch - only eleven new lines of code - which
> should have the same net effect.

This patch looks good. I'll apply it. Can you provide an update to
t/robot/rules.t as well?

Regards,
Gisle

Re: problem w/ robotrules record parse

writes:

> I've added a "warn" line in the case where a record separation is assumed... see
> http://www.geocities.com/mvaneerde/RobotRules.patch-4.txt
>
> rules.t patch:
> http://www.geocities.com/mvaneerde/rules-patch.txt

These patches have now been applied. Thanks!

--Gisle