Python metaclasses: Why isn't __setattr__ call

问题:

I have the following python code:

class FooMeta(type):
    def __setattr__(self, name, value):
        print name, value
        return super(FooMeta, self).__setattr__(name, value)

class Foo(object):
    __metaclass__ = FooMeta
    FOO = 123
    def a(self):
        pass

I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.

What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.

回答1:

A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.

This:

class Foo(object):
    __metaclass__ = FooMeta
    FOO = 123
    def a(self):
        pass

Comes out pretty much as if you'd written:

d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
    pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)

Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).

The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.

---

Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:

from collections import MutableMapping


class SingleAssignDict(MutableMapping):
    def __init__(self, *args, **kwargs):
        self._d = dict(*args, **kwargs)

    def __getitem__(self, key):
        return self._d[key]

    def __setitem__(self, key, value):
        if key in self._d:
            raise ValueError(
                'Key {!r} already exists in SingleAssignDict'.format(key)
            )
        else:
            self._d[key] = value

    def __delitem__(self, key):
        del self._d[key]

    def __iter__(self):
        return iter(self._d)

    def __len__(self):
        return len(self._d)

    def __contains__(self, key):
        return key in self._d

    def __repr__(self):
        return '{}({!r})'.format(type(self).__name__, self._d)


class RedefBlocker(type):
    @classmethod
    def __prepare__(metacls, name, bases, **kwargs):
        return SingleAssignDict()

    def __new__(metacls, name, bases, sad):
        return super().__new__(metacls, name, bases, dict(sad))


class Okay(metaclass=RedefBlocker):
    a = 1
    b = 2


class Boom(metaclass=RedefBlocker):
    a = 1
    b = 2
    a = 3

Running this gives me:

Traceback (most recent call last):
  File "/tmp/redef.py", line 50, in <module>
    class Boom(metaclass=RedefBlocker):
  File "/tmp/redef.py", line 53, in Boom
    a = 3
  File "/tmp/redef.py", line 15, in __setitem__
    'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict

Some notes:

1. __prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
2. type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
3. I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
4. The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.

---

Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.

Compare Python 2:

class Foo(object):
    __metaclass__ = FooMeta
    FOO = 123
    def a(self):
        pass

Being roughly equivalent to:

d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
    pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)

Where the metaclass to invoke is determined from the dictionary, versus Python 3:

class Foo(metaclass=FooMeta):
    FOO = 123
    def a(self):
        pass

Being roughly equivalent to:

d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
    pass
d['a'] = a
Foo = FooMeta('Foo', (), d)

Where the dictionary to use is determined from the metaclass.

回答2:

There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:

class FooMeta(type):
    def __new__(self, name, bases, attrs):
        print attrs
        return type.__new__(self, name, bases, attrs)

class Foo(object):
    __metaclass__ = FooMeta
    FOO = 123

Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.

回答3:

Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().

It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.

回答4:

During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.

Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.