I'm making a web-crawler. For finding the links in a page I was using xpath in selenium
driver = webdriver.Firefox()
driver.get(side)
Listlinker = driver.find_elements_by_xpath("//a")
This worked fine. Testing the crawler however, I found that not all links come under the a tag. href is sometimes used in area or div tags as well.
Right now I'm stuck with
driver = webdriver.Firefox()
driver.get(side)
Listlinkera = driver.find_elements_by_xpath("//a")
Listlinkerdiv = driver.find_elements_by_xpath("//div")
Listlinkerarea = driver.find_elements_by_xpath("//area")
which really puts the crawl in web-crawler.
I've tried xpath "//@href", but that doesn't work. I've also tried several ways to get all href url's in an efficient manner, both using beautiful soup and lxml, but so far, to no avail. I'm sorry I do not have any code to show for my efforts with beautiful soup and lxml, but as these proved useless, I deleted them, which isn't the smartest practice, I know. I have now started to save these unsuccessful attempts, for my own sake, if I ever want to try again, and want to know what went wrong the first time
Any help I could get on this would be greatly appreciated.
