I'm working with scrapy 1.1 . I want to add scrapy-fake-user-agent
Scrapy middleware that would rotate user agents seamlessly and randomly. User Agent strings are supplied by the fake-useragent
module.
following the directions from the site, I have:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
However when I run it I get:
ImportError: No module named scrapy_fake_useragent.middleware
What am I doing wrong?
edit: I opened a python shell in the virtualenv:
In[4]: import scrapy.downloadermiddlewares as x
In[5]: x.useragent.UserAgentMiddleware
Out[5]: scrapy.downloadermiddlewares.useragent.UserAgentMiddleware
In[7]: import scrapy_fake_useragent as x
In[8]: dir(x)
Out[8]: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
So 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware 'seems to be available but not 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware' ?