Re: RobotRules fails on user-agents with spaces

On Fri, 2005-10-14 at 10:37, Gisle Aas wrote:
> writes:
>
> > The problem... if I include a space in my robot's user agent, it
> > will fail to recognize robots.txt records targeted to my robot.
>
> You are not allowed to have space in the user agent name. See section
> "3.8 Product Tokens" of RFC 2616 [1]. Isn't it an option to just
> rename your spider to something that follows the spec?

Perhaps it would help if WWW::RobotRules were to warn/die when setting
an agent with a space in? An excellent message would be "RFC2616 forbids
spaces in an agent's names".

-Nigel