How to import members of all modules within a pack

Question

I am developing a package that has a file structure similar to the following:

test.py
package/
    __init__.py
    foo_module.py
    example_module.py

If I call import package in test.py, I want the package module to appear similar to this:

>>> vars(package)
mapping_proxy({foo: <function foo at 0x…}, {example: <function example at 0x…})

In other words, I want the members of all modules in package to be in package's namespace, and I do not want the modules themselves to be in the namespace. package is not a sub-package.

Let's say my files look like this:

foo_module.py:

def foo(bar):
    return bar

example_module.py:

def example(arg):
    return foo(arg)

test.py:

print(example('derp'))

How do I structure the import statements in test.py, example_module.py, and __init__.py to work from outside the package directory (i.e. test.py) and within the package itself (i.e. foo_module.py and example_module.py)? Everything I try gives Parent module '' not loaded, cannot perform relative import or ImportError: No module named 'module_name'.

Also, as a side-note (as per PEP 8): "Relative imports for intra-package imports are highly discouraged. Always use the absolute package path for all imports. Even now that PEP 328 is fully implemented in Python 2.5, its style of explicit relative imports is actively discouraged; absolute imports are more portable and usually more readable."

I am using Python 3.3.

Answer 1

I want the members of all modules in package to be in package's namespace, and I do not want the modules themselves to be in the namespace.

I was able to do that by adapting something I've used in Python 2 to automatically import plug-ins to also work in Python 3.

In a nutshell, here's how it works:

1. The package's __init_.py file imports all the other Python files in the same package directory that don't start with an '_' (underscore) character.

2. It then adds any names in the imported module's namespace to that of __init__ module's (which is also the package's namespace). Note I had to make the example_module module explicitly import foo from the foo_module.

One important aspect of doing things this way is realizing that it's dynamic and doesn't require the package module names to be hardcoded into the __init__.py file. Of course this requires more code to accomplish, but also makes it very generic and able to work with just about any (single-level) package — since it will automatically import new modules when they're added and no longer attempt to import any removed from the directory.

test.py:

from package import *

print(example('derp'))

__init__.py:

def _import_all_modules():
    """ Dynamically imports all modules in this package. """
    import traceback
    import os
    global __all__
    __all__ = []
    globals_, locals_ = globals(), locals()

    # Dynamically import all the package modules in this file's directory.
    for filename in os.listdir(__name__):
        # Process all python files in directory that don't start
        # with underscore (which also prevents this module from
        # importing itself).
        if filename[0] != '_' and filename.split('.')[-1] in ('py', 'pyw'):
            modulename = filename.split('.')[0]  # Filename sans extension.
            package_module = '.'.join([__name__, modulename])
            try:
                module = __import__(package_module, globals_, locals_, [modulename])
            except:
                traceback.print_exc()
                raise
            for name in module.__dict__:
                if not name.startswith('_'):
                    globals_[name] = module.__dict__[name]
                    __all__.append(name)

_import_all_modules()

foo_module.py:

def foo(bar):
    return bar

example_module.py:

from package.foo_module import foo  # added

def example(arg):
    return foo(arg)

Answer 2

I think you can get the values you need without cluttering up your namespace, by using from module import name style imports. I think these imports will work for what you are asking for:

Imports for example_module.py:

from package.foo_module import foo

Imports for __init__.py:

from package.foo_module import foo
from package.example_module import example

__all__ = [foo, example] # not strictly necessary, but makes clear what is public

Imports for test.py:

from package import example

Note that this only works if you're running test.py (or something else at the same level of the package hierarchy). Otherwise you'd need to make sure the folder containing package is in the python module search path (either by installing the package somewhere Python will look for it, or by adding the appropriate folder to sys.path).