{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 SAY GOODBYE 的问题《How to web crawl some sites [closed]》','https://www.manongdao.com/q-1156117.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
I am starting a new project of crawling websites to retrieve and store data internally using a web service. I looked up some information and came across Scrapy and Beevolve web crawling services.
My question is is it best to just create my own crawler with no prior experience or rent a web crawling service?
One issue that I came across is, some of the websites require a log in before getting any data.
If you want to create your own web crawler in Java you may want to look at this
You could also take a look at jSpider and jsoup.
Edit : This could work too : crawler4j