Python lazy evaluator

Question

Is there a Pythonic way to encapsulate a lazy function call, whereby on first use of the function f(), it calls a previously bound function g(Z) and on the successive calls f() returns a cached value?

Please note that memoization might not be a perfect fit.

I have:

f = g(Z)
if x:
     return 5
elif y:
     return f
elif z:
     return h(f)

The code works, but I want to restructure it so that g(Z) is only called if the value is used. I don't want to change the definition of g(...), and Z is a bit big to cache.

EDIT: I assumed that f would have to be a function, but that may not be the case.

Answer 1

Try using this decorator:

class Memoize:
    def __init__ (self, f):
        self.f = f
        self.mem = {}
    def __call__ (self, *args, **kwargs):
        if (args, str(kwargs)) in self.mem:
            return self.mem[args, str(kwargs)]
        else:
            tmp = self.f(*args, **kwargs)
            self.mem[args, str(kwargs)] = tmp
            return tmp

(extracted from dead link: http://snippets.dzone.com/posts/show/4840 / https://web.archive.org/web/20081026130601/http://snippets.dzone.com/posts/show/4840) (Found here: Is there a decorator to simply cache function return values? by Alex Martelli)

EDIT: Here's another in form of properties (using __get__) http://code.activestate.com/recipes/363602/

Answer 2

I'm a bit confused whether you seek caching or lazy evaluation. For the latter, check out the module lazy.py by Alberto Bertogli.

Answer 3

Even after your edit, and the series of comments with detly, I still don't really understand. In your first sentence, you say the first call to f() is supposed to call g(), but subsequently return cached values. But then in your comments, you say "g() doesn't get called no matter what" (emphasis mine). I'm not sure what you're negating: Are you saying g() should never be called (doesn't make much sense; why does g() exist?); or that g() might be called, but might not (well, that still contradicts that g() is called on the first call to f()). You then give a snippet that doesn't involve g() at all, and really doesn't relate to either the first sentence of your question, or to the comment thread with detly.

In case you go editing it again, here is the snippet I am responding to:

I have:

a = f(Z)
if x:
     return 5
elif y:
     return a
elif z:
     return h(a)

The code works, but I want to restructure it so that f(Z) is only called if the value is used. I don't want to change the definition of f(...), and Z is a bit big to cache.

If that is really your question, then the answer is simply

if x:
    return 5
elif y:
    return f(Z)
elif z:
    return h(f(Z))

That is how to achieve "f(Z) is only called if the value is used".

I don't fully understand "Z is a bit big to cache". If you mean there will be too many different values of Z over the course of program execution that memoization is useless, then maybe you have to resort to precalculating all the values of f(Z) and just looking them up at run time. If you can't do this (because you can't know the values of Z that your program will encounter) then you are back to memoization. If that's still too slow, then your only real option is to use something faster than Python (try Psyco, Cython, ShedSkin, or hand-coded C module).

Answer 4

You can employ a cache decorator, let see an example

from functools import wraps

class FuncCache(object):
    def __init__(self):
        self.cache = {}

    def __call__(self, func):
        @wraps(func)
        def callee(*args, **kwargs):
            key = (args, str(kwargs))
            # see is there already result in cache
            if key in self.cache:
                result = self.cache.get(key)
            else:
                result = func(*args, **kwargs)
                self.cache[key] = result
            return result
        return callee

With the cache decorator, here you can write

my_cache = FuncCache()

@my_cache
def foo(n):
    """Expensive calculation

    """
    sum = 0
    for i in xrange(n):
        sum += i
    print 'called foo with result', sum
    return sum

print foo(10000)
print foo(10000)
print foo(1234)

As you can see from the output

called foo with result 49995000
49995000
49995000

The foo will be called only once. You don't have to change any line of your function foo. That's the power of decorators.

Answer 5

There are quite a few decorators out there for memoization:

http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize http://code.activestate.com/recipes/498110-memoize-decorator-with-o1-length-limited-lru-cache/ http://code.activestate.com/recipes/496879-memoize-decorator-function-with-cache-size-limit/

Coming up with a completely general solution is harder than you might think. For instance, you need to watch out for non-hashable function arguments and you need to make sure the cache doesn't grow too large.

If you're really looking for a lazy function call (one where the function is only actually evaluated if and when the value is needed), you could probably use generators for that.

EDIT: So I guess what you want really is lazy evaluation after all. Here's a library that's probably what you're looking for:

http://pypi.python.org/pypi/lazypy/0.5

Answer 6

Here's a pretty brief lazy-decorator, though it lacks using @functools.wraps (and actually returns an instance of Lazy plus some other potential pitfalls):

class Lazy(object):
    def __init__(self, calculate_function):
        self._calculate = calculate_function

    def __get__(self, obj, _=None):
        if obj is None:
            return self
        value = self._calculate(obj)
        setattr(obj, self._calculate.func_name, value)
        return value


# Sample use:

class SomeClass(object):

    @Lazy
    def someprop(self):
        print 'Actually calculating value'
        return 13


o = SomeClass()
o.someprop
o.someprop