I'm trying to use PhantomJS/CasperJS to scrape a webpage. I've spent the last few days reading the docs, and searching online, but I'm stuck.
The page I'm scraping shows three levels of links - years, months, and days. When you select a Year, Month, and day, a count appears in the #count div. Also, the months are actually inputs that change an image in the #imageLoad div (which I don't need).
<div id="years">
<span class="year">2010</span>
<span class="year">2011</span>
<span class="year">2012</span>
etc...
</div>
<div id="months">
<input type="image" class="month" src="jan_image.png" onclick="changepic('jan')" />
<input type="image" class="month" src="feb_image.png" onclick="changepic('feb')" />
<input type="image" class="month" src="mar_image.png" onclick="changepic('mar')" />
etc...
</div>
<div id="days">
<span class="day">1</span>
<span class="day">2</span>
<span class="day">3</span>
etc...
</div>
<div id="imageLoad">
</div>
<div id="count">
</div>
I'm trying to loop through the links and record the count that appears for each combination of years, months, and days. As you can see, the months are inputs that change the picture.
I tried a number of things. The main thing I want to do is a nested loop that loops through each set of links, clicking them as I go. Here is the code (I'm using jQuery):
casper.start(link);
casper.then(function() {
pageInfo = this.evaluate(function(){
values = [];
for(var y = 0; y < $('#years').length; y++){
year= $('#years span').get(y);
$(year).click();
for(var m = 0; m < $('#months').length; m++){
month= $('#months input').get(m);
$(month).click();
for(var d = 0; d < $('#days').length; d++){
day= $('#days span').get(d);
$(day).click();
values.push($('#count').text());
}
}
}
return values;
});
});
This I thought would loop through each set of links in order, and I would get all the values from every variation of year, month and day.
However, when I click on the month inputs in my script, the script breaks and goes to the next casper.then statement. Is there a better way for me to do this?
I have the feeling that I'm going about this the wrong way, but nothing else I've tried has been fruitful either. It always seems like once it breaks to the next "then" there's no coming back to my loop.
I've tried looping with Casper.each
, but I don't know how many elements there will be before hand.
Thanks in advance.