I have a simple goal: load webpages with either phantom.js
(out of the box) or casper.js
(nice and easier) but using proxy and rotate it from a list if current one is bad (i.e. webpage loads fail or something like that).
I know casper.js has --proxy
param but it dictates the user to specify only ONE proxy and use it during runtime.
Question #1 is: how to rotate proxy on the fly programmatically?
I did some research and found this node-requester but it's not integrated with casper.js
. I tried to extract out just the proxy feature in the code but didn't really understand how it works in the nutshell (I am not that smart I guess).
So question #2: is there some simple implementation of proxy rotation that works with either phantom.js
or casper.js
?
I prefer to use the fancy casper.js
though but will go down with phantom.js
bare as well.
I had the same issue a while back, I worked with PhantomJS. The solution we ended up with was running PhantomJS as a child process of a larger Java/Scala server which then handled failures and assigned the different proxies when needed (by rerunning with different params in the --proxy arg).