Difference between BeautifulSoup and Scrapy crawle

问题:

I want to make a website that shows the comparison between amazon and e-bay product price. Which of these will work better and why? I am somewhat familiar with BeautifulSoup but not so much with Scrapy crawler.

回答1:

Scrapy is a Web-spider or web scraper framework, You give Scrapy a root URL to start crawling, then you can specify constraints on how many (number of) URLs you want to crawl and fetch,etc. It is a complete framework for web-scraping or crawling.

While

BeautifulSoup is a parsing library which also does a pretty good job of fetching contents from URL and allows you to parse certain parts of them without any hassle. It only fetches the contents of the URL that you give and then stops. It does not crawl unless you manually put it inside an infinite loop with certain criteria.

In simple words, with Beautiful Soup you can build something similar to Scrapy. Beautiful Soup is a library while Scrapy is a complete framework.

source: http://www.quora.com/Python-programming-language-1/How-is-BeautifulSoup-different-from-Scrapy

回答2:

I think both are good... im doing a project right now that use both. First i scrap all the pages using scrapy and save that on a mongodb collection using their pipelines, also downloading the images that exists on the page. After that i use BeautifulSoup4 to make a pos-processing where i must change attributes values and get some special tags.

If you don't know which pages products you want, a good tool will be scrapy since you can use their crawlers to run all amazon/ebay website looking for the products without making a explicit for loop.

Take a look at the scrapy documentation, it's very simple to use.

回答3:

The way I do it is to use the eBay/Amazon API's rather than scrapy, and then parse the results using BeautifulSoup.

The APIs gives you an official way of getting the same data that you would have got from scrapy crawler, with no need to worry about hiding your identity, mess about with proxies,etc.

回答4:

Both are using to parse data.

Scrapy:

• Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.
• But it has some limitations when data comes from java script or loading dynamicaly, we can over come it by using packages like splash, selenium etc.

BeautifulSoup:

• Beautiful Soup is a Python library for pulling data out of HTML and XML files.

• we can use this package for getting data from java script or dynamically loading pages.

Scrapy with BeautifulSoup is one of the best combo we can work with for scraping static and dynamic contents

回答5:

Using scrapy you can save tons of code and start with structured programming, If you dont like any of the scapy's pre-written methods then BeautifulSoup can be used in the place of scrapy method. Big project takes both advantages.

回答6:

The differences are many and selection of any tool/technology depends on individual needs.

Few major differences are:

1. BeautifulSoup is comparatively is easy to learn than Scrapy.
2. The extensions, support, community is larger for Scrapy than for BeautifulSoup.
3. Scrapy should be considered as a Spider while BeautifulSoup is a Parser.

回答7:

BeautifulSoup is only used to parse HTML and extract data,

Scrapy is used to download HTML, process data and save it.

For Detailed comparison see this