Is there a web crawler library available for PHP o

Question

Is there a web crawler library available for PHP or Ruby? a library that can do it depth first or breadth first... and handle the links even when href="../relative_path.html" and base url is used.

Answer 1

If you need to scrape web pages that use javascript you can use Capybara with a driver which will spin up a real browser, such as poltergeist. Its usually used with a testing framework for acceptance testing, but can also be used outside a testing framework.

Answer 2

If you'd like to learn basic web crawler & search things, you can start look at "luna engine".

Answer 3

http://phpcrawl.cuab.de/

Answer 4

Check this page out for a Ruby library: Ruby Mechanize

I'd like to mention that you would still be responsible for the way in which your crawler traverses sites.

Answer 5

you can go for webrat or watir in ruby, much easier than mechanize