Sibling package imports

Question

I've tried reading through questions about sibling imports and even the package documentation, but I've yet to find an answer.

With the following structure:

├── LICENSE.md
├── README.md
├── api
│   ├── __init__.py
│   ├── api.py
│   └── api_key.py
├── examples
│   ├── __init__.py
│   ├── example_one.py
│   └── example_two.py
└── tests
│   ├── __init__.py
│   └── test_one.py

How can the scripts in the examples and tests directories import from the api module and be run from the commandline?

Also, I'd like to avoid the ugly sys.path.insert hack for every file. Surely this can be done in Python, right?

Answer 1

First, you should avoid having files with the same name as the module itself. It may break other imports.

When you import a file, first the interpreter checks the current directory and then searchs global directories.

Inside examples or tests you can call:

from ..api import api

Answer 2

Seven years after

Since I wrote the answer below, modifying sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements

• Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using setup.cfg to store the metadata)
• Using the -m flag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
• For the tests, specifically, pytest is able to find the api package in this situation and takes care of the sys.path hacks for you

So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through pip -e is probably your best bet, even if it is not perfect yet.

Old answer

As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a __main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of

The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern.

(here)

Though, I use this pattern on a regular basis with

# Ugly hack to allow absolute import from the root folder
# whatever its name is. Please forgive the heresy.
if __name__ == "__main__" and __package__ is None:
    from sys import path
    from os.path import dirname as dir

    path.append(dir(path[0]))
    __package__ = "examples"

import api

Here path[0] is your running script's parent folder and dir(path[0]) your top level folder.

I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example api's parent folder).

Answer 3

You don't need and shouldn't hack sys.path unless it is necessary and in this case it is not. Use:

import api.api_key # in tests, examples

Run from the project directory: python -m tests.test_one.

You should probably move tests (if they are api's unittests) inside api and run python -m api.test to run all tests (assuming there is __main__.py) or python -m api.test.test_one to run test_one instead.

You could also remove __init__.py from examples (it is not a Python package) and run the examples in a virtualenv where api is installed e.g., pip install -e . in a virtualenv would install inplace api package if you have proper setup.py.

Answer 4

Just in case someone using Pydev on Eclipse end up here: you can add the sibling's parent path (and thus the calling module's parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g. from sibling import some_class.

Answer 5

I don't yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For examples or tests to import stuff from ..\api, it would look like:

import sys.path
import os.path
# Import from sibling directory ..\api
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..")
import api.api
import api.api_key

Answer 6

Tired on sys.path hacks?

There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand: The setuptools. I am not sure if there are edge cases which do not work well with this. The following is tested with Python 3.6.5, (Anaconda, conda 4.5.1), Windows 10 machine.

---

Setup

The starting point is the file structure you have provided, wrapped in a folder called myproject.

.
└── myproject
    ├── api
    │   ├── api_key.py
    │   ├── api.py
    │   └── __init__.py
    ├── examples
    │   ├── example_one.py
    │   ├── example_two.py
    │   └── __init__.py
    ├── LICENCE.md
    ├── README.md
    └── tests
        ├── __init__.py
        └── test_one.py

I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.

api.py

As a test case, let's use the following ./api/api.py

def function_from_api():
    return 'I am the return value from api.api!'

test_one.py

from api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

Try to run test_one:

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\myproject\tests\test_one.py", line 1, in <module>
    from api.api import function_from_api
ModuleNotFoundError: No module named 'api'

Also trying relative imports wont work:

Using from ..api.api import function_from_api would result into

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\tests\test_one.py", line 1, in <module>
    from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package

---

Steps

1) Make a setup.py file to the root level directory

The contents for the setup.py would be*

from setuptools import setup, find_packages

setup(name='myproject', version='1.0', packages=find_packages())

2) Use a virtual environment

If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)

• Create virtual env
  • python -m venv venv
• Activate virtual env
  • . /venv/bin/activate (Linux) or ./venv/Scripts/activate (Win)

To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.

Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis

PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>

and your folder tree should look like this**

.
├── myproject
│   ├── api
│   │   ├── api_key.py
│   │   ├── api.py
│   │   └── __init__.py
│   ├── examples
│   │   ├── example_one.py
│   │   ├── example_two.py
│   │   └── __init__.py
│   ├── LICENCE.md
│   ├── README.md
│   └── tests
│       ├── __init__.py
│       └── test_one.py
├── setup.py
└── venv
    ├── Include
    ├── Lib
    ├── pyvenv.cfg
    └── Scripts [87 entries exceeds filelimit, not opening dir]

3) pip install your project in editable state

Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.

In the root directory, run

pip install -e . (note the dot, it stands for "current directory")

You can also see that it is installed by using pip freeze

(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
  Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0

4) Add myproject. into your imports

Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the setup.py & pip install will work still work fine. See an example below.

---

Test the solution

Now, let's test the solution using api.py defined above, and test_one.py defined below.

test_one.py

from myproject.api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

running the test

(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!

---

* See the setuptools docs for more verbose setup.py examples.

** In reality, you could put your virtual environment anywhere on your hard disk.