Using Nutch how to crawl the dynamic content of we

Question

I am using apache Nutch 1.10 to crawl the web pages and to extract the contents in the page. Some of the links contains dynamic contents which are loaded on the call of ajax. Nutch cannot able to crawl and extract the dynamic contents of ajax. How can I solve this? Is there any solution? if yes please help me with your answers.

Thanks in advance.

Answer 1

Checkout the latest Nutch 1.11 trunk which includes a new plugin protocol-interactive selenium. (https://github.com/apache/nutch/tree/trunk/src/plugin/protocol-interactiveselenium)

This plugin allows you to write your own handler and execute javascript to get dynamic content.

Answer 2

Most web crawler libraries do not offer javascript rendering out of the box. You usually have to plugin another library or product that offers js rendering like Selenium or PhantomJS.

Here is a tutorial using nutch and Selenium.