What is the best way to do web scripting/web macro

Question

I'm trying to streamline some of our tasks at my place of work, and it seems that quite a lot of our developers' time is spent doing semi-mechanical tasks on the web (specifically, editing online stores that use web-based interfaces). As such, I've been looking into some solutions that will allow these tasks to be done by scripts since I figure that could save us quite a bit of time per task. So before I really started digging into any of these, I was just wondering if the Stack Overflow community had any recommendations about which web scripting/macro solution would be the best.

Here are the requirements:

• Must be able to interact with web forms (not just downloading a page and scraping the file - the script must edit controls within a web form and then submit that form)
• The forms we have to edit are secure forms, so the scripting solution must be able to handle that (ie it's of no use for us to have an incredibly powerful scripting solution if a human being will have to sit there and watch it and manually re-login every few minutes)
• It would be really, really, really preferable if it could read local files and do some basic string replacement/manipulation on them (e.g. it would be nice to have a list of variables or some HTML code in a text file and then have the script replace the token "STORENAME" with that particular store's name before it inserts the code into the form)

Here are the scripting solutions that are on my radar so far (I haven't really looked into any of these, although I have played around with Chickenfoot):

• Chickenfoot
• iMacros for Firefox
• libwww-perl
• libwww for unix and C (while searching for libwww for perl I came across this, which I did not know existed until now)
• a more "general" macro solution like AutoHotKey

Has anyone here on Stack Overflow tried any of these solutions? If so, what did you like or dislike about them? Can anyone recommend one that is not on the list? (This is by no means an exclusive or exhaustive list). I would really love to automate a lot of our mechanical processes, and I hope the Stack Overflow community can help us out so we can hopefully avoid that much of the mind-numbing part of the work :).

edit: Re: platform - We have primarily WindowsXP terminals at work, but 1) we do have a few Mac test PCs, so OS X is a viable option, and 2) if it would mean automating a lot of these tasks, I'll build a Linux box if that is necessary. So platform is pretty much a non-issue.

Answer 1

my impression of IMacros is you need to pay to unlock a lot of the goodies, so I went with Chickenfoot. Unfortunately the Chickenfoot mailing list is very quiet lately, so am not confident about its future...

Answer 2

I'm a big fan of Selenium

http://www.seleniumhq.com

designed to handle automated browser testing though with scripting support (multiple languages) you can design macro templates that will fill in forms and such based on variables

also: anyone know a good macro that works with Flash?? Doesn't seem javascript is able to cross that line

Answer 3

WatiN, .net version of Watir. I like it a lot.

Answer 4

I'm actually a big fan of Chickenfoot, which I have started using a lot recently to provide scripting capabilities for Zotero, another Firefox extension.

Answer 5

Use:

Firefox + iMacros to fetch:
Go to the page you like (e.g yahoo.com), bring iMacro in focus, hit record and search for some text. When first page shows, Save the page in a file.

Now you have a page you wanted.
You are done fetching at this point.
Save the macro and give it a name. Next time you can just run the macro you just saved.

You then clean up the html code you got by using this:

# this is php, do the proper thing in the language of your choice
# one or more spaces is equal to one space, remove tab, return  ...etc.
return ereg_replace(' +', ' ', trim(preg_replace("/(\r\n|\r|\t|\n|\'|\")/s", '', $string)

Now you have a clean html page with one line, a long one.
Now you can cut and slice as you wish by using a regEx.

Things to know:
You can start the macro in firefox via a cronjob (commandline) if you need full automation.

Answer 6

we use the iMacros tool you mentioned with good success:

Firefox + iMacros

Note that it also comes as free IE addon! The great thing about this is that the exactly same macros work in both browsers. IMHO very useful for quick and easy web testing and general web automation.

Tim