Selecting an option from an optgroup with selenium

Question

I would like to choose a value from this optgroup which should then give me a dropdown of links.

<div class="searchbar">
    <select id="q" multiple="" tabindex="-1" class="select2-hidden-accessible" aria-hidden="true">
    <option></option>
    <option class="q-all-text" value="al:all">Search all text</option>

    <optgroup label="Business Type">
        <option value="bt:Buyer">Buyer</option>
        <option value="bt:Farmer/Rancher">Farmer/Rancher</option>
        <option value="bt:Farmers Market">Farmers Market</option>
        <option value="bt:Fishery">Fishery</option>
        <option value="bt:Food Bank">Food Bank</option>
    </optgroup>

Here is my code so far:

driver = webdriver.Chrome('/path/to/chromedriver')
driver.get("https://foodmarketmaker.com/main/mmsearch")
iframe = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(iframe)

select  = Select(driver.find_element_by_class_name("select2-hidden- accessible"))
select.select_by_value("bt:Farmer/Rancher")
links = driver.find_elements_by_tag_name('a')
print(links)
for link in links:
    print(link.get_attribute('href'))

I either get an exception that the element does not exist, or when I access by index, i get an ElementNotVisibleException due to the aria-hidden attribute being true. Is there any way around this?

Answer 1

As per the HTML you have shared to print all the options you have to induce WebDriverWait to switch to the desired frame, then click on the Search Box for the options to be visible as follows :

• Code Block :

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  
  options = Options()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  options.add_argument("--disable-extensions")
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://foodmarketmaker.com/main/mmsearch")
  WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@src='https://search.foodmarketmaker.com']")))
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='searchbar']//input[@class='select2-search__field' and @placeholder='start typing to search']"))).click()
  links = driver.find_elements_by_xpath("//li[@class='select2-results__option' and @aria-label='Business Type']/strong//following::ul[1]//li")
  for link in links:
      print(link.get_attribute('innerHTML'))
  

• Console Output :

  Buyer
  Buyer: Baked Goods
  Buyer: Beverages
  Buyer: Dairy
  Buyer: Fish & Seafood
  Buyer: Fruit & Nuts
  Buyer: Grains & OilSeed
  Buyer: Herbs
  Buyer: Meat & Poultry
  Buyer: Miscellaneous
  Buyer: Specialty Products
  Buyer: Sugar/Confectionary & Snacks
  Buyer: Vegetables
  Eating & Drinking Place
  Eating & Drinking Place: Cafeteria & Buffet
  Eating & Drinking Place: Caterer
  Eating & Drinking Place: Drinking Place
  Eating & Drinking Place: Food Service Contractor
  Eating & Drinking Place: Mobile Food Service
  Eating & Drinking Place: Restaurant
  Eating & Drinking Place: Snack & Nonalcoholic Beverage Bar
  Farmer/Rancher
  Farmer/Rancher: Dairy
  Farmer/Rancher: Forage
  Farmer/Rancher: Fruits & Nuts
  Farmer/Rancher: Grains
  Farmer/Rancher: Herbs
  Farmer/Rancher: Livestock Production (Beef Cattle)
  Farmer/Rancher: Meat & Poultry
  Farmer/Rancher: Specialty Products
  Farmer/Rancher: Vegetables
  Farmers Market
  Fishery
  Fishery: Fish/Shellfish/Seafood
  Food Bank
  Food Bank: Food Bank
  Food Hub
  Food Retailer
  Food Retailer: Baked Goods Store
  Food Retailer: Beer, Wine & Liquor Store
  Food Retailer: Candy, Nut & Confectionery
  Food Retailer: Fish Market
  Food Retailer: Fruit & Vegetable Market
  Food Retailer: Grocery/Convenience Store
  Food Retailer: Health Supplement Store
  Food Retailer: Meat & Poultry Market
  Food Retailer: Other Direct Selling Establishments
  Food Retailer: Other Specialty Food Store
  General Member
  Processor/Packing Shed
  Processor/Packing Shed: Animal Food Manufacturing
  Processor/Packing Shed: Bakery Products
  Processor/Packing Shed: Beverages
  Processor/Packing Shed: Dairy Products
  Processor/Packing Shed: Fish/Shellfish/Seafood Products
  Processor/Packing Shed: Fruit & Vegetable Products
  Processor/Packing Shed: Grain & Oilseed Milling
  Processor/Packing Shed: Meat & Poultry Products
  Processor/Packing Shed: Miscellaneous
  Processor/Packing Shed: Packing Shed
  Processor/Packing Shed: Sugar and Confectionery Products
  Tourism
  Tourism: Agritourism
  Tourism: Fishing Charter
  Tourism: Hunting
  Wholesaler
  Wholesaler: Beer, Wine & Alcoholic Beverages
  Wholesaler: Confectionery & Snacks
  Wholesaler: Dairy Products
  Wholesaler: Farm Products
  Wholesaler: Fish/Shellfish/Seafood
  Wholesaler: Fruit & Vegetable
  Wholesaler: General Grocery
  Wholesaler: Meat & Poultry
  Wholesaler: Other Grocery
  Wholesaler: Packaged Frozen Food
  Winery
  Winery: Blends or Generic Wine
  Winery: Fortified Wine
  Winery: Fruit Wine
  Winery: Other Wine Products
  Winery: Red Wine
  Winery: Sparkling Wine
  Winery: White Wine
  

Answer 2

select is not the node that you need to handle as it's not visible.

Try below code to get required output:

from selenium import webdriver as web
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


driver = web.Chrome('/path/to/chromedriver')
driver.get("https://foodmarketmaker.com/main/mmsearch")
iframe = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(iframe)

wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//section[@id="search-right"]//input[@placeholder="start typing to search"]'))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//li[.="Bass"]'))).click()

links = wait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//section[@id="search-results"]//a[.//*[name()="svg"]]')))

for link in links:
    print(link.get_attribute('href'))

P.S. You need to replace text value of li node in '//li[.="Bass"]' XPath with required option