I have a set of Scrapy spiders. They need to be run daily from a desktop application. What is the simplest way (from user's point of view) to install and run it on another windows machine?
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试):
问题:
回答1:
The simplest way is write a script in python for them I guess...
If you are running a Windows Server you can even schedule the comand that you use (scrapy crawl yoursprider) to run the spiders.
回答2:
Create a script (e.g. run_spider.py) which runs scrapy crawl <spider_name>
as a system command.
run_spider.py
from os import system
output_file_name = 'results.csv'
system('scrapy crawl myspider -o ' + output_file_name + ' -t csv')
Then feed that script to PyInstaller:
pyinstaller run_spider.py
回答3:
Here is another possibility to run your spider as standalone script or executable
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
You can find more information here: https://doc.scrapy.org/en/1.0/topics/practices.html