get a browser rendered html+javascript

问题:

I need a comandline tool (or Javascript/PHP, but i think commandline is the one way) for render and get the rendered content of URL, but the important its I need to renderer the Javascript not only the CSS/Html/images.

For example command like: "renderengine http://www.google.es outputfile.html" and the content of the web (parsed html and javascript executed) isa saved in outputfile.html.

I need this because i need to take the result of a full javascript website like grooveshark, the site load all using javascript/ajax and the crawlers dont find nothing, only basic HTML empty template (because is loaded after using ajax/javscript)

Exists any browser engine for linux with support to Javascript (for example V8) that output the result for save in files?

回答1:

Try phantomjs from www.phantomjs.org and you can easily modify the included rasterize.js to export the rendered HTML. It's based on webkit and does full evaluation of your target site's javascript, allowing you to adjust timeouts or execute your own code first if you wish. I personally use it to save hardcopy HTML file version of fully-rendered knockout.js templates.

It executes javascript so I just did something like this and saved the console output to a file:

var markup = page.evaluate(function(){return document.documentElement.innerHTML;});
console.log(markup);
phantom.exit();

回答2:

• PhantomJS (first suggested by nvuono) : can export the rendered page as non-HTML (pdf, png...). Closely related: SlimerJS, CasperJS
• Xvfb a display server implementing the X11 display server protocol, without showing any screen output. Alternative: XDummy
• HTtrack : command-line tool
• Selenium : very complete solution with bindings in many languages
• puppeteer : headless Chrome API, usable in NodeJS or as a command-line tool
• Apache Notch & webmagic : open source Java web crawlers
• pholcus : "distributed & high concurrency" web crawler written in Go

And there are many Python web scrapping libraries:

• Scrapy
• pyspider
• ghost.py
• splinter