We're trying to scrape the course names, qualification and duration of the course and store each in a separate array. With the below we pull all of that, but it seems to be in random order, with some parts potentially ordered by page etc. Wondering if anybody is able to help.
require 'mechanize'
mechanize = Mechanize.new
@duration_array = []
@qual_array = []
@courses_array = []
page = mechanize.get('http://search.ucas.com/search/results?Vac=2&AvailableIn=2016&IsFeatherProcessed=True&page=1&providerids=41')
page.search('div.courseinfoduration').each do |x|
puts x.text.strip
page.search('div.courseinfooutcome').each do |y|
puts y.text.strip
end
while next_page_link = page.at('.pager a[text()=">"]')
page = mechanize.get(next_page_link['href'])
page.search('div.courseinfoduration').each do |x|
name = x
@duration_array.push(name)
puts x.text.strip
end
end
while next_page_link = page.at('.pager a[text()=">"]')
page = mechanize.get(next_page_link['href'])
page.search('div.courseinfooutcome').each do |y|
name = y
@qual_array.push(name)
puts y.text.strip
end
end
page.search('div.coursenamearea h4').each do |h4|
puts h4.text.strip
end
while next_page_link = page.at('.pager a[text()=">"]')
page = mechanize.get(next_page_link['href'])
page.search('div.coursenamearea h4').each do |h4|
name = h4.text
@courses_array.push(name)
puts h4.text.strip
end
end
end