How to import all submodules?

Question

I have a directory structure as follows:

| main.py
| scripts
|--| __init__.py
   | script1.py
   | script2.py
   | script3.py

From main.py, the module scripts is imported. I tried using pkgutils.walk_packages in combination with __all__, but using that, I can only import all the submodules directly under main using from scripts import *. I would like to get them all under scripts. What would be the cleanest way to import all the submodules of scripts so that I could access scripts.script1 from main?

EDIT: I am sorry that I was a bit vague. I would like to import the submodules on run-time without specifying them explicitly in __init__.py. I can use pkgutils.walk_packages to get the submodule names (unless someone knows of a better way), but I am not sure of the cleanest way to use these names (or maybe the ImpImporters that walk_packages returns?) to import them.

Answer 1

Simply works, and allows relative import inside packages:

def import_submodules(package_name):
    """ Import all submodules of a module, recursively

    :param package_name: Package name
    :type package_name: str
    :rtype: dict[types.ModuleType]
    """
    package = sys.modules[package_name]
    return {
        name: importlib.import_module(package_name + '.' + name)
        for loader, name, is_pkg in pkgutil.walk_packages(package.__path__)
    }

Usage:

__all__ = import_submodules(__name__).keys()

Answer 2

Not nearly as clean as I would like, but none of the cleaner methods worked for me. This achieves the specified behaviour:

Directory structure:

| pkg
|--| __init__.py
   | main.py
   | scripts
   |--| __init__.py
      | script1.py
      | script2.py
      | script3.py

Where pkg/scripts/__init__.py is empty, and pkg/__init__.py contains:

import importlib as _importlib
import pkgutil as _pkgutil
__all__ = [_mod[1].split(".")[-1] for _mod in
           filter(lambda _mod: _mod[1].count(".") == 1 and not 
                               _mod[2] and __name__ in _mod[1],
                  [_mod for _mod in _pkgutil.walk_packages("." + __name__)])]
__sub_mods__ = [".".join(_mod[1].split(".")[1:]) for _mod in
                filter(lambda _mod: _mod[1].count(".") > 1 and not 
                                    _mod[2] and __name__ in _mod[1],
                       [_mod for _mod in 
                        _pkgutil.walk_packages("." + __name__)])]
from . import *
for _module in __sub_mods__:
    _importlib.import_module("." + _module, package=__name__)

Although it's messy, it should be portable. I've used this code for several different packages.

Answer 3

This works nicely for me in Python 3.3. Note that this works only for submodules which are in files in the same directory as the __init__.py. With some work however it can be enhanced for supporting submodules in directories too.

from glob import iglob
from os.path import basename, relpath, sep, splitext

def import_submodules(__path__to_here):
    """Imports all submodules.
    Import this function in __init__.py and put this line to it:
    __all__ = import_submodules(__path__)"""
    result = []
    for smfile in iglob(relpath(__path__to_here[0]) + "/*.py"):
        submodule = splitext(basename(smfile))[0]
        importstr = ".".join(smfile.split(sep)[:-1])
        if not submodule.startswith("_"):
            __import__(importstr + "." + submodule)
            result.append(submodule)
    return result