Web Crawler - Ignore Robots.txt file?

Question

Some servers have a robots.txt file in order to stop web crawlers from crawling through their websites. Is there a way to make a web crawler ignore the robots.txt file? I am using Mechanize for python.

Answer 1

This looks like what you need:

from mechanize import Browser
br = Browser()

# Ignore robots.txt
br.set_handle_robots( False )

but you know what you're doing…

Answer 2

The documentation for mechanize has this sample code:

br = mechanize.Browser()
....
# Ignore robots.txt.  Do not do this without thought and consideration.
br.set_handle_robots(False)

That does exactly what you want.