Handling pagination in lxml

问题:

I am trying to mirror a ruby scraper that I wrote but for a python only environment. I've decided to use lxml and requests to get this done. My problem is pagination:

base_url = "http://example.com/something/?page=%s"  
for url in [base_url % i for i in xrange(10)]:  
    r = requests.get(url)

I'm new to python and this library so I'm not sure the best way to perform the equivalent ruby code:

last_pg = (page.xpath("//div[contains(@class, 'b-tabs-utility')]").text.split('of ')[-1].split(' Results')[0].to_f / 60).ceil
puts "Current Index: #{last_pg}"
for pg_number in 1..last_pg do
  puts "Getting links on page #{pg_number}"

回答1:

Get the number of results by splitting the b-tabs-utility div into spaces and getting element before last:

base_url = "http://example.com/something/?page=%d" 
results_per_page = 60

div = page.xpath("//div[contains(@class, 'b-tabs-utility')]")[0].text
last_pg = int(div.split()[-2]) / results_per_page
for i in xrange(last_pg):
    r = requests.get(base_url % i)

I'm assuming the div text is in the following format, example: ... of 642 Results