Is it possible to make grequests and requests_cach

Question

Look at this code:

import requests
import grequests
import requests_cache
requests_cache.install_cache('bla')

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
results = grequests.map(rs)

I would expect that after executing this I will find bla.sqlite file in the current directory and executing

results = grequests.map(rs)

will be MUCH faster because data will be taken from sqlite cache. Unfortunately this is not true, file wasn't created at all and there is no speedup. When I use requests insetead of grequests everything works fine. So the question is a title says: Is it possible to make grequests and requests_cache work together? and if yes, how?

Answer 1

The requests_cache.install_cache() function patches requests.Session, but you already imported grequests, which used:

from requests import Session

As such, grequests never uses the patched session object.

Move the import to after you installed the cache:

import requests_cache
requests_cache.install_cache('bla')
import grequests

Alternatively, create a CachedSession object and pass that in to the grequests.get() (and related) methods as the session parameter:

import grequests
import requests_cache

session = requests_cache.CachedSession('bla')

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u, session=session) for u in urls)
results = grequests.map(rs)

Take into account that the cache storage backend may not be able to handle the corcurrent access safely. The sqlite backend uses a thread lock, for example, which might well clash.