Web Crawlers

I am looking for a newsgroup or forum that discusses best practices in
using methods to influence how web crawlers record a website.

Information will be appreciated,
Stan Hilliard

Re: Web Crawlers

Stan Hilliard wrote in
news:bhdb635dtmeeqsgd37q343l8plsro6kl33@4ax.com:

> I am looking for a newsgroup or forum that discusses best practices in
> using methods to influence how web crawlers record a website.
>
> Information will be appreciated,
> Stan Hilliard
>

http://www.robotstxt.org/wc/faq.html

Re: Web Crawlers

On Tue, 05 Jun 2007 20:34:58 GMT, Fuzzy Logic
wrote:

>Stan Hilliard wrote in
>news:bhdb635dtmeeqsgd37q343l8plsro6kl33@4ax.com:
>
>> I am looking for a newsgroup or forum that discusses best practices in
>> using methods to influence how web crawlers record a website.
>>
>> Information will be appreciated,
>> Stan Hilliard
>>
>
>http://www.robotstxt.org/wc/faq.html

Thanks. I have placed a robots.txt on my website. But I notice from my
log files that many web crawlers do not even check it. Can some web
crawlers be malicious?

Is there a free tool that can analyze a web site to determine its
vulnerabilities?

Stan Hilliard

Re: Web Crawlers

On Wed, 13 Jun 2007 08:08:00 -0500, Stan Hilliard
wrote:

>On Tue, 05 Jun 2007 20:34:58 GMT, Fuzzy Logic
> wrote:
>
>>Stan Hilliard wrote in
>>news:bhdb635dtmeeqsgd37q343l8plsro6kl33@4ax.com:
>>
>>> I am looking for a newsgroup or forum that discusses best practices in
>>> using methods to influence how web crawlers record a website.
>>>
>>> Information will be appreciated,
>>> Stan Hilliard
>>>
>>
>>http://www.robotstxt.org/wc/faq.html
>
>Thanks. I have placed a robots.txt on my website. But I notice from my
>log files that many web crawlers do not even check it. Can some web
>crawlers be malicious?
>
>Is there a free tool that can analyze a web site to determine its
>vulnerabilities?
>
>Stan Hilliard

My web site is on a remote windows 2000 server running IIS & MS
Frontpage. It is hosted by an internet service provider. I am mostly
interested in whether permissions are properly set, what
vulnerabilities it has, and occurrences or clues of malicious
intrusion.

Stan Hilliard