Overriding the default type() metaclass before Pyt

Question

Here be dragons. You've been warned.

I'm thinking about creating a new library that will attempt to help write a better test suite.
In order to do that one of the features is a feature that verifies that any object that is being used which isn't the test runner and the system under test has a test double (a mock object, a stub, a fake or a dummy). If the tester wants the live object and thus reduce test isolation it has to specify so explicitly.

The only way I see to do this is to override the builtin type() function which is the default metaclass.
The new default metaclass will check the test double registry dictionary to see if it has been replaced with a test double or if the live object was specified.

Of course this is not possible through Python itself:

>>> TypeError: can't set attributes of built-in/extension type 'type'

Is there a way to intervene with Python's metaclass lookup before the test suite will run (and probably Python)?
Maybe using bytecode manipulation? But how exactly?

Answer 1

The following is not advisable, and you'll hit plenty of problems and cornercases implementing your idea, but on Python 3.1 and onwards, you can hook into the custom class creation process by overriding the __build_class__ built-in hook:

import builtins


_orig_build_class = builtins.__build_class__


class SomeMockingMeta(type):
    # whatever


def my_build_class(func, name, *bases, **kwargs):
    if not any(isinstance(b, type) for b in bases):
        # a 'regular' class, not a metaclass
        if 'metaclass' in kwargs:
            if not isinstance(kwargs['metaclass'], type):
                # the metaclass is a callable, but not a class
                orig_meta = kwargs.pop('metaclass')
                class HookedMeta(SomeMockingMeta):
                    def __new__(meta, name, bases, attrs):
                        return orig_meta(name, bases, attrs)
                kwargs['metaclass'] = HookedMeta
            else:
                # There already is a metaclass, insert ours and hope for the best
                class SubclassedMeta(SomeMockingMeta, kwargs['metaclass']):
                    pass
                kwargs['metaclass'] = SubclassedMeta
        else:
            kwargs['metaclass'] = SomeMockingMeta

    return _orig_build_class(func, name, *bases, **kwargs)


builtins.__build_class__ = my_build_class

This is limited to custom classes only, but does give you an all-powerful hook.

For Python versions before 3.1, you can forget hooking class creation. The C build_class function directly uses the C-type type() value if no metaclass has been defined, it never looks it up from the __builtin__ module, so you cannot override it.

Answer 2

I like your idea, but I think you're going slightly off course. What if the code calls a library function instead of a class? Your fake type() would never be called and you would never be advised that you failed to mock that library function. There are plenty of utility functions both in Django and in any real codebase.

I would advise you to write the interpreter-level support you need in the form of a patch to the Python sources. Or you might find it easier to add such a hook to PyPy's codebase, which is written in Python itself, instead of messing with Python's C sources.

I just realized that the Python interpreter includes a comprehensive set of tools to enable any piece of Python code to step through the execution of any other piece of code, checking what it does down to each function call, or even to each single Python line being executed, if needed.

sys.setprofile should be enough for your needs. With it you can install a hook (a callback) that will be notified of every function call being made by the target program. You cannot use it to change the behavior of the target program, but you can collect statistics about it, including your "mock coverage" metric.

Python's documentation about the Profilers introduces a number of modules built upon sys.setprofile. You can study their sources to see how to use it effectively.

If that turns out not to be enough, there is still sys.settrace, a heavy-handed approach that allows you to step through every line of the target program, inspect its variables and modify its execution. The standard module bdb.py is built upon sys.settrace and implements the standard set of debugging tools (breakpoints, step into, step over, etc.) It is used by pdb.py which is the commandline debugger, and by other graphical debuggers.

With these two hooks, you should be all right.