Change python mro at runtime

Question

I've found myself in an unusual situation where I need to change the MRO of a class at runtime.

The code:

class A(object):
    def __init__(self):
        print self.__class__
        print "__init__ A"
        self.hello()

    def hello(self):
        print "A hello"

class B(A):
    def __init__(self):
        super(B, self).__init__()
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        print "%s hello" % self.msg_str

a = A()
b = B()

As to be expected, this fails as the __init__ method of A (when called from B) calls B's hello which attempts to access an attribute before it exists.

The issue is that I'm constrained in the changes I can make:

• B must subclass A
• A cannot be changed
• Both A and B require a hello method
• B cannot initialise other attributes before calling the super __init__

I did solve this conceptually, by changing the MRO at runtime. In brief, during B's __init__, but before calling super __init__, the MRO would be changed so that A would be searched for methods first, thereby calling A's hello instead of B's (and therefore failing).

The issue is that MRO is read only (at class runtime).

Is there another way to implement this ? Or possibly a different solution altogether (that still respects the aforementioned constraints) ?

Answer 1

The other provided answers are advisable if you are not bound by the constraints mentioned in the question. Otherwise, we need to take a journey into mro hacks and metaclass land.

After some reading, I discovered you can change the mro of a class, using a metaclass.

This however, is at class creation time, not at object creation time. Slight modification is necessary.

The metaclass provides the mro method, which we overload, that is called during class creation (the metaclass' __new__ call) to produce the __mro__ attribute.

The __mro__ attribute is not a normal attribute, in that:

1. It is read only
2. It is defined BEFORE the metaclass' __new__ call

However, it appears to be recalculated (using the mro method) when a class' base is changed. This forms the basis of the hack.

In brief:

• The subclass (B) is created using a metaclass (change_mro_meta). This metaclass provides:
  • An overloaded mro method
  • Class methods to change the __mro__ attribute
  • A class attribute (change_mro) to control the mro behaviour

As mentioned, modifying the mro of a class while in its __init__ is not thread safe.

The following may disturb some viewers. Viewer discretion is advised.

The hack:

class change_mro_meta(type):
    def __new__(cls, cls_name, cls_bases, cls_dict):
        out_cls = super(change_mro_meta, cls).__new__(cls, cls_name, cls_bases, cls_dict)
        out_cls.change_mro = False
        out_cls.hack_mro   = classmethod(cls.hack_mro)
        out_cls.fix_mro    = classmethod(cls.fix_mro)
        out_cls.recalc_mro = classmethod(cls.recalc_mro)
        return out_cls

    @staticmethod
    def hack_mro(cls):
        cls.change_mro = True
        cls.recalc_mro()

    @staticmethod
    def fix_mro(cls):
        cls.change_mro = False
        cls.recalc_mro()

    @staticmethod
    def recalc_mro(cls):
        # Changing a class' base causes __mro__ recalculation
        cls.__bases__  = cls.__bases__ + tuple()

    def mro(cls):
        default_mro = super(change_mro_meta, cls).mro()
        if hasattr(cls, "change_mro") and cls.change_mro:
            return default_mro[1:2] + default_mro
        else:
            return default_mro

class A(object):
    def __init__(self):
        print "__init__ A"
        self.hello()

    def hello(self):
        print "A hello"

class B(A):
    __metaclass__ = change_mro_meta
    def __init__(self):
        self.hack_mro()
        super(B, self).__init__()
        self.fix_mro()
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        print "%s hello" % self.msg_str

a = A()
b = B()

Some notes:

The hack_mro, fix_mro and recalc_mro methods are staticmethods to the metaclass but classmethods to the class. It did this, instead of multiple inheritance, because I wanted to group the mro code together.

The mro method itself returns the default ordinarily. Under the hack condition, it appends the second element of the default mro (the immediate parent class) to the mro, thereby causing the parent class to see its own methods first before the subclass'.

I'm unsure of the portability of this hack. Its been tested on 64bit CPython 2.7.3 running on Windows 7 64bit.

Don't worry, I'm sure this won't end up in production code somewhere.

Answer 2

There may be grander solutions but a simple option is to write class B defensively. For example:

class B(A):
    def __init__(self):
        super(B, self).__init__()
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        if not hasattr(self, 'msg_str'):
            A.hello(self)
            return
        print "%s hello" % self.msg_str

A good editor with regex capability could auto-insert appropriate if not hasattr(self, 'some_flag'):... lines as the first lines of any methods in B.

Answer 3

I'd like to point out a solution which is very specific to the example you present in your question, and therefor unlikely to help. (But in case it does help at all...)

You can bypass hello's polymorphism by defining it as a class member, instead of a method.

class B(A):
    def __init__(self):
        super(B, self).__init__()
        print "__init__ B"
        self.msg_str = "B"
        self.hello = lambda: print "%s hello" % self.msg_str
        self.hello()

(A remains unchanged).

This solution will break if:

• you subclass B and need to override hello in the subclass
• msg_str is modified after __init__ runs
• probably several other cases...

Answer 4

For your particular example, one solution is to give B a class attribute to hold a default message:

class B(A):

    msg_str = "default msg"

    def __init__(self):
        super(B, self).__init__()
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        print "%s hello" % self.msg_str

Usually this causes confusion, but in this case it might be helpful. If B.hello is called before the instance's msg_str is set, it will read the class one. Once the instance msg_str is set, it shadows the class one so that future accesses of self.msg_str will see the instance-specific one.

I don't quite understand why you can't set attributes before calling the superclass __init__. Depending on exactly what the underlying situation is, there may be other solutions.

Answer 5

My solution would be to ask for forgiveness:

class A(object):
    def __init__(self):
        print self.__class__
        print "__init__ A"
        self.hello()

    def hello(self):
        print "A hello"

class B(A):
    def __init__(self):
        super(B, self).__init__()
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        try:
            print "%s hello" % self.msg_str
        except AttributeError:
            pass  # or whatever else you want

a = A()
b = B()

or if you do not want to refactor methods called from init:

class A(object):
    def __init__(self):
        print self.__class__
        print "__init__ A"
        self.hello()

    def hello(self):
        print "A hello"

class B(A):
    def __init__(self):
        try:
            super(B, self).__init__()
        except AttributeError:
            pass  # or whatever else you want
        print "__init__ B"
        self.msg_str = "B"
        self.hello()

    def hello(self):
        print "%s hello" % self.msg_str

a = A()
b = B()

Answer 6

I don't know if it's relevant to the specific problem, but it seems to me that changing the MRO on the fly like that could be risky in a concurrent program, and could definitely have issues if any of these objects turn out to be created recursively.

A non-MRO-based solution occurs to me, depending on the nature of the errors this code would have encountered. (Keeping in mind that this is belated. Perhaps somebody else will want a different answer.)

Basically, each hello() method on B would be wrapped in a decorator. Something along the lines of

class deferring(object):

    def __init__(self, function):
        self.function = function

    def __get__(self, instance, owner):
        # Return an unbound method, or whatever, when called from B.
        if instance is None:
            return self.function.__get__(None, owner)
        else:
            # Indicate that an instance is ready via a flag.
            # May need to tweak this based on the exact problem.
            if hasattr(instance, '_set_up'):
                return self.function.__get__(instance, owner)
            else:
                # Walk the mro manually.
                for cls in owner.__mro__:
                    # Crazy inefficient. Possible to mitigate, but risky.
                    for name, attr in vars(cls).items():
                        if attr is self:
                            break
                    else:
                        continue
                    return getattr(super(cls, instance), name)
                else:
                    raise TypeError

If you don't want to go the descriptor route, it's also possible to do something like

def deferring(function):
    def wrapped(self, *args, **kwargs):
        if hasattr(self, '_set_up'):
            return function(self, *args, **kwargs)
        else:
            for cls in type(self).__mro__:
                for name, attr in vars(cls).items():
                    if attr is function:
                        break
                else:
                    continue
                return getattr(super(cls, self), name)(*args, **kwargs)
            else:
                raise TypeError
    return wrapped