{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 傲 的问题《Database for web crawler in python?》','https://www.manongdao.com/q-548195.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
Hi im writing a web crawler in python to extract news articles from news websites like nytimes.com. i want to know what would be a good db to use as a backend for this project?
Thanks in advance!
相关问题
- how to define constructor for Python's new Nam
- streaming md5sum of contents of a large remote tar
- How to get the background from multiple images by
- Evil ctypes hack in python
- Correctly parse PDF paragraphs with Python
This could be a great project to use a document database like CouchDB, MongoDB, or SimpleDB.
MongoDB has a hosted solution: http://mongohq.com. There is also a binding for Python (Pymongo).
SimpleDB is a great choice if you are hosting this on Amazon Web Services
CouchDB is an open source package from the Apache Foundation.
You can take a look at Firebird
Firebird python driver are developped by the core team
Personally, I love PostGreSQL -- but other free DBs such as MySql (or, if you have reasonably small amounts of data -- a few GB at most -- even the SQLite that comes with Python) will be fine too.
I think the database itself will probably be one of the easier aspects of a web crawler like this.
If expect high load reading or writing the database (for example if you intend to run many crawlers at the same time) then you will want to steer in the direction of MySql, otherwise something like Sqlite will probably do you just fine.