Pandas memoization

Question

I have lengthy computations which I repeat many times. Therefore, I would like to use memoization (packages such as jug and joblib), in concert with Pandas. The problem is whether the package would memoize well Pandas DataFrames as method arguments.

Has anyone tried it? Is there any other recommended package/way to do this?

Answer 1

Author of jug here: jug works fine. I just tried the following and it works:

from jug import TaskGenerator
import pandas as pd
import numpy as np


@TaskGenerator
def gendata():
    return pd.DataFrame(np.arange(343440).reshape((10,-1)))

@TaskGenerator
def compute(x):
    return x.mean()

y = compute(gendata())

It is not as efficient as it could be as it just uses pickle internally for the DataFrame (although it compresses it on the fly, so it is not horrible in terms of memory use; just slower than it could be).

I would be open to a change which saves these as a special case as jug currently does for numpy arrays: https://github.com/luispedro/jug/blob/master/jug/backends/file_store.py#L102

Answer 2

I use this basic memoization decorator, memoized. http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize

DataFrames are hashable, so it should work fine. Here's an example.

In [2]: func = lambda df: df.apply(np.fft.fft)

In [3]: memoized_func = memoized(func)

In [4]: df = DataFrame(np.random.randn(1000, 1000))

In [5]: %timeit func(df)
10 loops, best of 3: 124 ms per loop

In [9]: %timeit memoized_func(df)
1000000 loops, best of 3: 1.46 us per loop

Looks good to me.