robots.txt with httpd and root level SetHandler perl-script

Hi,

I'd like to have a robots.txt on a site that has the following
apache httpd config:


SetHandler perl-script
PerlHandler RT::Mason


But if I install a robots.txt to the DocumentRoot and test it via wget
I just download the front page of the site, as its handled by
perl-script. It it possible to have a robots.txt in this situation?

thanks for any tips, Andy.

Re: robots.txt with httpd and root level SetHandler perl-script

a.smith@ukgrid.net wrote:
> Hi,
>
> I'd like to have a robots.txt on a site that has the following apache
> httpd config:
>
>
> SetHandler perl-script
> PerlHandler RT::Mason
>
>
> But if I install a robots.txt to the DocumentRoot and test it via wget I
> just download the front page of the site, as its handled by perl-script.
> It it possible to have a robots.txt in this situation?
>
> thanks for any tips, Andy.
>
Ideas :
1) Try a section inside the above section, to reset the handler
to the default Apache (that may not be so easy)
2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"

Re: robots.txt with httpd and root level SetHandler perl-script

> a.smith@ukgrid.net wrote:
> > Hi,
> >
> > I'd like to have a robots.txt on a site that has the following apache
> > httpd config:
> >
> >
> > SetHandler perl-script
> > PerlHandler RT::Mason
> >
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget I
> > just download the front page of the site, as its handled by perl-script.
> > It it possible to have a robots.txt in this situation?
> >
> > thanks for any tips, Andy.
> >
> Ideas :
> 1) Try a section inside the above section, to reset the handler
> to the default Apache (that may not be so easy)

It may be easier to have the files match only the needed suffixes
(e.g., .pl, .perl, .html, etc.), otherwise you may need to exclude
other types in the future such as .css, .js, .png, .jpg, .jpeg, .gif,
..ico, etc.

> 2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"

Be careful here -- you may need to return it with the correct
content type of "text/plain" since "as-is" may be for raw file output
(including HTTP headers).

Randolf Richardson - randolf@inter-corporate.com
Inter-Corporate Computer & Network Services, Inc.
Vancouver, British Columbia, Canada
http://www.inter-corporate.com

Re: robots.txt with httpd and root level SetHandler perl-script

On Monday, March 14, 2011 14:53:35 Andr=E9 Warnier wrote:
> > I'd like to have a robots.txt on a site that has the following apache=
=20
> >
> > httpd config:
> >=20
> >
> > SetHandler perl-script
> > PerlHandler RT::Mason
> >
> >=20
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget =
I=20
> > just download the front page of the site, as its handled by perl-script=
.
> > It it possible to have a robots.txt in this situation?
> >
> >=20
> >
> > thanks for any tips, Andy.
> >
> >=20
>=20
> Ideas :
> 1) Try a section inside the above section, to
> reset the handler to the default Apache (that may not be so easy)

SetHandler None (imho)

> 2) Create a Mason handler to handle the URL "robots.txt" and return the
> file "as is"

or instead of the Location block:

PerlMapToStorageHandler "sub { \
use Apache2::Const -compile=3D>DECLINED; \
use Apache2::RequestRec (); \
use Apache2::RequestUtil (); \
unless( $_[0]->uri eq '/robots.txt' ) { \
$_[0]->add_config(['SetHandler perl-script', \
'PerlHandler RT::Mason']); \
} \
return Apache2::Const::DECLINED; \
}"

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: robots.txt with httpd and root level SetHandler perl-script

Quoting Torsten Förtsch :

> or instead of the Location block:
>
> PerlMapToStorageHandler "sub { \
> use Apache2::Const -compile=>DECLINED; \
> use Apache2::RequestRec (); \
> use Apache2::RequestUtil (); \
> unless( $_[0]->uri eq '/robots.txt' ) { \
> $_[0]->add_config(['SetHandler perl-script', \
> 'PerlHandler RT::Mason']); \
> } \
> return Apache2::Const::DECLINED; \
> }"

Thanks Torsten (and all). So in this situation do I put this code in
httpd.conf in between the lines? Just to check,
as you said "instead of the Location block"?

thanks Andy.

Re: robots.txt with httpd and root level SetHandler perl-script

On Monday, March 14, 2011 18:45:39 a.smith@ukgrid.net wrote:
> Quoting Torsten Förtsch :
> > or instead of the Location block:
> >=20
> > PerlMapToStorageHandler "sub { \
> >
> > use Apache2::Const -compile=3D>DECLINED; \
> > use Apache2::RequestRec (); \
> > use Apache2::RequestUtil (); \
> > unless( $_[0]->uri eq '/robots.txt' ) { \
> > $_[0]->add_config(['SetHandler perl-script', \
> > 'PerlHandler RT::Mason']); \
> > } \
> > return Apache2::Const::DECLINED; \
> >
> > }"
>=20
> Thanks Torsten (and all). So in this situation do I put this code in =20
> httpd.conf in between the lines? Just to check, =20
> as you said "instead of the Location block"?

A PerlMapToStorageHandler inside a Location block doesn't make sense.

See
http://perl.apache.org/docs/2.0/user/config/config.html#mod_ perl_Directives=
_Argument_Types_and_Allowed_Location

BTW, it should read

use Apache2::Const -compile=3D>'DECLINED';

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: robots.txt with httpd and root level SetHandler perl-script

Quoting Torsten Förtsch :

>
> A PerlMapToStorageHandler inside a Location block doesn't make sense.
>
> See
> http://perl.apache.org/docs/2.0/user/config/config.html#mod_ perl_Directives_Argument_Types_and_Allowed_Location
>
> BTW, it should read
>
> use Apache2::Const -compile=>'DECLINED';
>

Hi,

so, if I have other blocks, will setting as you suggest
override all of them? That would break other parts of the httpd
virtual host on my config...

thanks Andy.

Re: robots.txt with httpd and root level SetHandler perl-script

On Monday, March 14, 2011 19:14:35 a.smith@ukgrid.net wrote:
> if I have other blocks, will setting as you suggest =20
> override all of them? That would break other parts of the httpd =20
> virtual host on my config...

no, Location blocks are evaluated *after* a PerlMapToStorageHandler. You ca=
n=20
also try


SetHandler None


Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net