googlebot indexes links it finds in newsgroups - just for your info

You probably know this already, but 8 hours after publishing a testing area
link in this newsgroup, googlebot goes straight for it - it checked
robots.txt first, which I hadn't set up to exclude this folder.

Re: googlebot indexes links it finds in newsgroups - just for your info

On 24 Sep 2007 in alt.html, Joe Butler said:

> You probably know this already, but 8 hours after publishing a testing
> area link in this newsgroup, googlebot goes straight for it - it
> checked robots.txt first, which I hadn't set up to exclude this
> folder.
>
>
>
>

Yes it seems to spring like a panther on new domain names too, the
strange and unpredictable beast.

--

Free Website Promotion and Marketing tips: http://tinyurl.com/2f5lb8

Re: googlebot indexes links it finds in newsgroups - just for your info

"CW Northrop" wrote in message
news:Xns99B520ACF5B60cwnorthrop@216.196.109.145...
> On 24 Sep 2007 in alt.html, Joe Butler said:
>
>> You probably know this already, but 8 hours after publishing a testing
>> area link in this newsgroup, googlebot goes straight for it - it
>> checked robots.txt first, which I hadn't set up to exclude this
>> folder.
>>
>>
>>
>>
>
> Yes it seems to spring like a panther on new domain names too, the
> strange and unpredictable beast.
>

LOL

+mrcakey