Scripting HTTP more effeciently

Question

Often times I want to automate http queries. I currently use Java(and commons http client), but would probably prefer a scripting based approach. Something really quick and simple. Where I can set a header, go to a page and not worry about setting up the entire OO lifecycle, setting each header, calling up an html parser... I am looking for a solution in ANY language, preferable scripting

Answer 1

My turn : wget or perl with lwp. You'll find example on the linked page.

Answer 2

Have a look at Selenium. It generates code for C#, Java, Perl, PHP, Python, and Ruby if you need to customize the script.

Answer 3

Some ruby libraries:

• httparty: really interesting, the philosophy is interesting.
• mechanize: classic good-quality web automatization library.
• scrubYt: puzzling at first glance but fun to use.

Answer 4

Watir sounds close to what you want although it (like Selenium linked to in another answer) actually opens up a browser to do stuff. You can see some examples here. Another browser based record + playback approach system is sahi.

If your application is uses WSGI, then paste is a nice option.

Mechanize linked to in another answer is a "browser in a library" and there are clones in perl, Ruby and Python. The Perl one is the original one and this seems to be the way to go if you don't want a browser. The problem with this approach is that all the front end code (which might rely on JavaScript), won't be exercised.

Answer 5

What about using PHP+Curl, or just bash?

Answer 6

Mechanize for Python seems easy to use: http://wwwsearch.sourceforge.net/mechanize/