Stop abusive bots from crawling?

Question

Is this a good idea??

http://browsers.garykeith.com/stream.asp?RobotsTXT

What does abusive crawling mean? How is that bad for my site?

Answer 1

Not really. Most "bad bots" ignore the robots.txt file anyway.

Abuse crawling usually means scraping. These bots are showing up to harvest email addresses or more commonly, content.

As to how you can stop them? That's really tricky and often not wise. Anti-crawl techniques have a tendency to be less than perfect and cause problems for regular humans.

Sadly, like "shrinkage" in retail, it's a cost of doing business on the web.

Answer 2

A user-agent (which includes crawlers) is under no obligation to honour your robots.txt. The best you can do is try to identify abusive access patterns (via web-logs, etc.), and block the corresponding IP.