I have around 6,00,000 entries in MongoDB
in the following format:
feature:category:count
where
- feature could be any word,
- category is positive or negative, and
- count tells how many times a feature occurred in a document for that category.
I want to cache the top 1000 tuples, let's say so as not to query database each time.
How does one build an LRU cache in Python? Or are there any known solutions to this?
Besides the version included in Python 3.2, there are LRU cache recipes in the Python Cookbook including these by Python core developer Raymond Hettinger.
The LRU cache in Python3.3 has O(1) insertion, deletion, and search.
The design uses a circular doubly-linked list of entries (arranged oldest-to-newest) and a hash table to locate individual links. Cache hits use the hash table to find the relevant link and move it to the head of the list. Cache misses delete the oldest link and create a new link at the head of the linked list.
Here's a simplified (but fast) version in 33 lines of very basic Python (using only simple dictionary and list operations). It runs on Python2.0 and later (or PyPy or Jython or Python3.x):
There are also backports of the python 3.3 version of lru_cache such as this which runs on python 2.7. If you are interested in two layers of caching (if not cached in the instance it will check a shared cache) I have created lru2cache based on the backport of lru_cache.
Python 3.2
functools
includes an LRU cache. You could easily cherrypick it from repo, check if you have to adjust it to work with Python 2 (shouldn't be too hard - perhaps usingitertools
instead of certain builtins - ask if you need help) and be done. You need to wrap the query it into a callable and make sure it depends on the (hashable) function arguments, though.