Re: WWW::RobotRules warning could be more helpful

njh@despammed.com writes:

> If you spider several sites and one of them has a broken robots.txt file you
> can't tell which one since the warning doesn't tell you.

This will be better in 5.801. I've applied a variation of Bill
Moseley's suggested patch for the same problem.

> Around line 73 of RobotRules.pm
> change:
> warn "RobotRules: Disallow without preceding User-agent\n";
> to
> # njh@bandsman.co.uk: added $netloc
> warn "RobotRules: $netloc Disallow without preceding User-agent\n";