How does __call__ actually work?

Question

Python's magic method __call__ is called whenever you attempt to call an object. Cls()() is thus equal to Cls.__call__(Cls()).

Functions are first class objects in Python, meaning they're just callable objects (using __call__). However, __call__ itself is a function, thus it too has __call__, which again has its own __call__, which again has its own __call__.

So Cls.__call__(Cls()) is thus equal to Cls.__call__.__call__(Cls()) and again equilevant to Cls.__call__.__call__.__call__(Cls()) and so on and so forth.

How does this infinite loop end? How does __call__ actually execute the code?

Answer 1

Under the hood, all calls in Python use the same mechanism, and almost all arrive at the same C function in the CPython implementation. Whether an object is an instance of a class with a __call__ method, a function (itself an object), or a builtin object, all calls (except for optimized special cases) arrive at the function PyObject_Call. That C function gets the object's type from the ob_type field of the object's PyObject struct, and then from the type (another PyObject struct) gets the tp_call field, which is a function pointer. If tp_call is not NULL, it calls through that, with the args and kwargs structures that were also passed to PyObject_Call.

When a class defines a __call__ method, that sets up the tp_call field appropriately.

Here's an article explaining all of this in detail: Python internals: How callables work. It even lists and explains the entire PyObject_Call function, which isn't very big. If you want to see that function in its native habitat, it's in Objects/abstract.c in the CPython repo.

Also relevant is this stackoverflow Q&A: What is a "callable" in Python?.

Answer 2

I didn't check any documentation, but from my tests it seem __call__ isn't always called:

def func1(*args, **kargs):
    print "func1 called", args, kargs

def func2(*args, **kargs):
    print "func2 called", args, kargs

func1.__call__ = func2

func1() # here is still called func1

class Cls:
    def __init__(*args, **kargs):
        print "init called", args, kargs
    def __call__(*args, **kargs):
        print "object called", args, kargs

obj = Cls() # here is actually called __init__
obj()  # here is called __call__

this prints

func1 called () {}
init called (<__main__.Cls instance at 0x0000000002A5ED88>,) {}
object called (<__main__.Cls instance at 0x0000000002A5ED88>,) {}

Answer 3

There isn't an actual infinite loop, because the __call__ method is not actually invoked ("called") for all of those situations. It's only invoked directly when there is a function-like call on an object that provides a __call__ method.

Normal class instantiation Cls(...) and regular functional invocation f() are known cases that are handled directly. There generally is not an actual invocation of __call__(), so there are a finite number of __call__ method invocations that can ever occur, even in complex cases with deep inheritance, metaclasses, etc.

Because there was some dispute as to whether the short-circuiting of conceptual infinite loops was really happening, let's look at the disassembled bytecode. Consider the following code:

def f(x):
    return x + 1

class Adder(object):
    def something(self, x):
        return x + 19
    def __call__(self, x):
        return x + 1

def lotsacalls(y):
    u = f(1)
    a = Adder()
    z = u + a.something(y)
    return a(z * 10)

Sorry it's a little complex, as I want to show several instances of short-circuiting--namely, normal def functions, __init__ calls, normal methods, and __call__ special methods. Now:

So here are a range of times when, if Python were really, truly "walking the tree" of conceptual __call__ invocations, it would to reference Function (and possibly Method classes, and invoke their __call__ methods). It doesn't. It uses the simple bytecode CALL_FUNCTION in all cases, short-circuiting the conceptual tree-walk down. Logically you can imagine that there is a class Function that has a __call__ method that's invoked when a function (i.e. an instance of the Function class) is called. But it doesn't really work that way. The compiler, bytecode interpreter, and other parts of the C-language underpinnings do not actually walk meta-class trees. They short-circuit like crazy.