How to add a third party Scrapy middleware

2019-05-31 13:06发布

enter image description here

I'm working with scrapy 1.1 . I want to add scrapy-fake-user-agent Scrapy middleware that would rotate user agents seamlessly and randomly. User Agent strings are supplied by the fake-useragent module.

following the directions from the site, I have:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

However when I run it I get:

ImportError: No module named scrapy_fake_useragent.middleware

What am I doing wrong?

edit: I opened a python shell in the virtualenv:

In[4]: import scrapy.downloadermiddlewares as x
In[5]: x.useragent.UserAgentMiddleware
Out[5]: scrapy.downloadermiddlewares.useragent.UserAgentMiddleware

In[7]: import scrapy_fake_useragent as x
In[8]: dir(x)
Out[8]: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

So 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware 'seems to be available but not 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware' ?

标签: python scrapy
0条回答
登录 后发表回答