Circular & nested imports in python

Question

I'm having some real headaches right now trying to figure out how to import stuff properly. I had my application structured like so:

main.py
util_functions.py
widgets/
 - __init__.py
 - chooser.py
 - controller.py

I would always run my applications from the root directory, so most of my imports would be something like this

from util_functions import *
from widgets.chooser import *
from widgets.controller import *
# ...

And my widgets/__init__.py was setup like this:

from widgets.chooser import Chooser
from widgets.controller import MainPanel, Switch, Lever

__all__ = [
  'Chooser', 'MainPanel', 'Switch', 'Lever',
]

It was working all fine, except that widgets/controller.py was getting kind of lengthy, and I wanted it to split it up into multiple files:

main.py
util_functions.py
widgets/
 - __init__.py
 - chooser.py
 - controller/
    - __init__.py
    - mainpanel.py
    - switch.py
    - lever.py

One of issues is that the Switch and Lever classes have static members where each class needs to access the other one. Using imports with the from ___ import ___ syntax that created circular imports. So when I tried to run my re-factored application, everything broke at the imports.

My question is this: How can I fix my imports so I can have this nice project structure? I cannot remove the static dependencies of Switch and Lever on each other.

Answer 1

This is covered in the official Python FAQ under How can I have modules that mutually import each other.

As the FAQ makes clear, there's no silvery bullet that magically fixes the problem. The options described in the FAQ (with a little more detail than is in the FAQ) are:

• Never put anything at the top level except classes, functions, and variables initialized with constants or builtins, never from spam import anything, and then the circular import problems usually don't arise. Clean and simple, but there are cases where you can't follow those rules.
• Refactor the modules to move the imports into the middle of the module, where each module defines the things that need to be exported before importing the other module. This can means splitting classes into two parts, an "interface" class that can go above the line, and an "implementation" subclass that goes below the line.
• Refactor the modules in a similar way, but move the "export" code (with the "interface" classes) into a separate module, instead of moving them above the imports. Then each implementation module can import all of the interface modules. This has the same effect as the previous one, with the advantage that your code is idiomatic, and more readable by both humans and automated tools that expect imports at the top of a module, but the disadvantage that you have more modules.

As the FAQ notes, "These solutions are not mutually exclusive." In particular, you can try to move as much top-level code as possible into function bodies, replace as many from spam import … statements with import spam as is reasonable… and then, if you still have circular dependencies, resolve them by refactoring into import-free export code above the line or in a separate module.

---

With the generalities out of the way, let's look at your specific problem.

Your switch.Switch and lever.Lever classes have "static members where each class needs to access the other one". I assume by this you mean they have class attributes that are initialized using class attributes or class or static methods from the other class?

---

Following the first solution, you could change things so that these values are initialized after import time. Let's assume your code looked like this:

class Lever:
    switch_stuff = Switch.do_stuff()
    # ...

You could change that to:

class Lever:
    @classmethod
    def init_class(cls):
        cls.switch_stuff = Switch.do_stuff()

Now, in the __init__.py, right after this:

from lever import Lever
from switch import Switch

… you add:

Lever.init_class()
Switch.init_class()

That's the trick: you're resolving the ambiguous initialization order by making the initialization explicit, and picking an explicit order.

---

Alternatively, following the second or third solution, you could split Lever up into Lever and LeverImpl. Then you do this (whether as separate lever.py and leverimpl.py files, or as one file with the imports in the middle):

class Lever:
    @classmethod
    def get_switch_stuff(cls):
        return cls.switch_stuff

from switch import Swift

class LeverImpl(Lever):
    switch_stuff = Switch.do_stuff()

Now you don't need any kind of init_class method. Of course you do need to change the attribute to a method—but if you don't like that, with a bit of work, you can always change it into a "class @property" (either by writing a custom descriptor, or by using @property in a metaclass).

---

Note that you don't actually need to fix both classes to resolve the circularity, just one. In theory, it's cleaner to fix both, but in practice, if the fixes are ugly, it may be better to just fix the one that's less ugly to fix and leave the dependency in the opposite direction alone.