Python + web scraping + scrapy : How to get the li

Question

I have to scrape all movies from this IMDb page : https://www.imdb.com/list/ls055386972/.

My approach is first to scrape all the values of <a href="/title/tt0068646/?ref_=ttls_li_tt" , i.e., to extract /title/tt0068646/?ref_=ttls_li_tt portions and then add 'https://www.imdb.com' to prepare the complete URL to the movie, i.e., https://www.imdb.com/title/tt0068646/?ref_=ttls_li_tt . But whenever I am giving response.xpath('//h3[@class]/a[@href]').extract() it is extracting the desired portion along with the movie title: [u'<a href="/title/tt0068646/?ref_=ttls_li_tt">The Godfather</a>', u'<a href="/title/tt0108052/?ref_=ttls_li_tt">Schindler\'s List</a>......]'I want only the "/title/tt0068646/?ref_=ttls_li_tt" portion.

How to proceed?

Answer 1

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.imdb.com/list/ls055386972/")
soup = BeautifulSoup(page.content, 'html.parser')

movies = soup.findAll('h3', attrs={'class' : 'lister-item-header'})
for movie in movies:
    print(movie.a['href'])

OUTPUT:

/title/tt0068646/?ref_=ttls_li_tt
/title/tt0108052/?ref_=ttls_li_tt
/title/tt0050083/?ref_=ttls_li_tt
/title/tt0118799/?ref_=ttls_li_tt
.
.
.
.
/title/tt0088763/?ref_=ttls_li_tt
/title/tt0266543/?ref_=ttls_li_tt

Answer 2

it is the working code please try:

class MoviesSpider():

  name = 'movies' #name of the spider
  allowed_domains = ['imdb.com']
  start_url = 'http://imdb.com/list/ls055386972/'

  def __init__(self):
      super(MoviesSpider, self).__init__()

  def start_requests(self):
      yield Request(self.start_url, callback=self.parse, headers=self.headers)

  def parse(self, response):
      #events = response.xpath('//*[@property="url"]/@href').extract()
      links = response.xpath('//h3[@class]/a/@href').extract()

      final_links = []

      for link in links:
          final_link = 'http://www.imdb.com' + link
          final_links.append(final_link)

      for final_link in final_links:
          absolute_url = response.urljoin(final_link)
          yield Request(absolute_url, callback = self.parse_movies)

          #process next page url
          #next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
          #absolute_next_page_url = response.urljoin(next_page_url)
          #yield Request(absolute_next_page_url)

  def parse_movies(self, response):

      title  = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()

      yield{
                'title': title,
      }

Answer 3

I would suggest you to use requests-html to get all the hyperlinks and remove the ones that doesn't match your criteria. You can even get the absolute url using r.html.absolute_links

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.imdb.com/list/ls055386972/')
links = r.html.links
for i in range(len(links)):
    if not links[i].startswith('/title/'):
        del links[i]
print(links)