{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 孤傲高冷的网名 的问题《How extract all URLs in a website using BeautifulS》','https://www.manongdao.com/q-1346419.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
I'm working on a project that require to extract all links from a website, with using this code I'll get all of links from single URL:
import requests
from bs4 import BeautifulSoup, SoupStrainer
source_code = requests.get('https://stackoverflow.com/')
soup = BeautifulSoup(source_code.content, 'lxml')
links = []
for link in soup.find_all('a'):
links.append(str(link))
problem is that if I want to extract all URLs, I have to write another for loop and then another one ... . I want to extract all URLs that are exist in this website and in this website's sub domains. is there any way to do this without writing nested for? and even with writing nested for, I don't know how many for should I use to get all URLs.
WoW, it takes about 30 min to find a solution, I found a simple and efficient way to do this, As @αԋɱҽԃ-αмєяιcαη mentioned, some time if your website linked to a BIG website like google, etc, it wont be stop until you memory get full of data. so there are steps that you should consider.
here a sample code and it should works fine, I actually tested it and it was fun fore me:
and the output will be:
I set the limitation to 162, you can increase it as many as you want and you ram allowed.
How's this?
You can also use this crawl frame, which can help you do many things
Well, actually what you are asking for is possible but that's mean an infinite loop which will keep run and run till your memory
BoOoOoOmAnyway the idea should be like the following.
you will use
for item in soup.findAll('a')andthen item.get('href')add to
setto get rid of duplicates urls and use withifconditionis not Noneto get rid ofNoneobjects.then keep looping over and over till your
setbecame0something likelen(urls)