soup.select('.r a') in f'https://googl

Question

The "I'm Feeling Lucky!" project in the "Automate the boring stuff with Python" ebook no longer works with the code he provided.

Specifically, the linkElems = soup.select('.r a')

I've already tried using the solution provided in: soup.select('.r a') in 'https://www.google.com/#q=vigilante+mic' gives empty list in python BeautifulSoup

, and I'm currently using the same search format.

import webbrowser, requests, bs4

def im_feeling_lucky():

    # Make search query look like Google's
    search = '+'.join(input('Search Google: ').split(" "))

    # Pull html from Google
    print('Googling...') # display text while downloading the Google page
    res = requests.get(f'https://google.com/search?q={search}&oq={search}')
    res.raise_for_status()

    # Retrieve top search result link
    soup = bs4.BeautifulSoup(res.text, features='lxml')


    # Open a browser tab for each result.
    linkElems = soup.select('.r')  # Returns empty list
    numOpen = min(5, len(linkElems))
    print('Before for loop')
    for i in range(numOpen):
        webbrowser.open(f'http://google.com{linkElems[i].get("href")}')

The linkElems variable returns an empty list [] and the program doesn't do anything past that.

Answer 1

I too had had the same problem while reading that book and found a solution for that problem.

replacing

soup.select('.r a')

with

soup.select('div#main > div > div > div > a')

will solve that issue

following is the code that will work

import webbrowser, requests, bs4 , sys

print('Googling...')
res = requests.get('https://google.com/search?q=' + ' '.join(sys.argv[1:]))
res.raise_for_status()

soup = bs4.BeautifulSoup(res.text)

linkElems = soup.select('div#main > div > div > div > a')  
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get("href"))

the above code takes input from commandline arguments

Answer 2

I took a different route. I saved the HTML from the request and opened that page, then I inspected the elements. It turns out that the page is different if I open it natively in the Chrome browser compared to what my python request is served. I identified the div with the class that appears to denote a result and supplemented that for the .r - in my case it was .kCrYT

#! python3

# lucky.py - Opens several Google Search results.

import requests, sys, webbrowser, bs4

print('Googling...') # display text while the google page is downloading

url= 'http://www.google.com.au/search?q=' + ' '.join(sys.argv[1:])
url = url.replace(' ','+')


res = requests.get(url)
res.raise_for_status()


# Retrieve top search result links.
soup=bs4.BeautifulSoup(res.text, 'html.parser')


# get all of the 'a' tags afer an element with the class 'kCrYT' (which are the results)
linkElems = soup.select('.kCrYT > a') 

# Open a browser tab for each result.
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open_new_tab('http://google.com.au' + linkElems[i].get('href'))