Here is my directory structure:
/home/dmugtasimov/tmp/name-res
root
tests
__init__.py
test_1.py
__init__.py
classes.py
extra.py
root.py
File contents: root/tests/_init_.py
import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__),
'../..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
sys.path.insert(0, ROOT_DIRECTORY)
# These imports are required for unittest to find test modules in package properly
from root.tests import test_1
root/tests/test_1.py
import unittest
from root.classes import Class1
class Tests(unittest.TestCase):
pass
root/_init_.py - empty
root/classes.py
import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
sys.path.insert(0, ROOT_DIRECTORY)
print 'sys.path:', sys.path
print 'BEFORE: import root.extra'
import root.extra
print 'AFTER: import root.extra'
class Class1(object):
pass
class Class2(object):
pass
root/extra.py
class Class3(object):
pass
root/root.py
import os
import sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
sys.path.insert(0, ROOT_DIRECTORY)
from classes import Class2
I get following output:
$ python -m unittest tests.test_1
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
main(module=None)
File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/usr/lib/python2.7/unittest/loader.py", line 128, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "tests/__init__.py", line 9, in <module>
from root.tests import test_1
File "/home/dmugtasimov/tmp/name-res/root/tests/__init__.py", line 9, in <module>
from root.tests import test_1
File "/home/dmugtasimov/tmp/name-res/root/tests/test_1.py", line 3, in <module>
from root.classes import Class1
File "/home/dmugtasimov/tmp/name-res/root/classes.py", line 9, in <module>
import root.extra
File "/home/dmugtasimov/tmp/name-res/root/root.py", line 6, in <module>
from classes import Class2
ImportError: cannot import name Class2
It turns out the problem is the order used by python interpreter to search for package or module:
$ python -vv -m unittest tests.test_1
…skipped...
import root.classes # precompiled from /home/dmugtasimov/tmp/name-res/root/classes.pyc
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
# trying /home/dmugtasimov/tmp/name-res/root/root.so
# trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so
# trying /home/dmugtasimov/tmp/name-res/root/root.py
# /home/dmugtasimov/tmp/name-res/root/root.pyc matches /home/dmugtasimov/tmp/name-res/root/root.py
…skipped...
According to python documentation http://docs.python.org/2/tutorial/modules.html#the-module-search-path: “When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path.”
This means that python should look at sys.path index 0 entry, get path '/home/dmugtasimov/tmp/name-res' and find package named root and then search module named extra in this package. But instead it searches in /home/dmugtasimov/tmp/name-res/root/ directory for module root and then tries to find something named extra in it. What does it happen? Does not it contradict official documentation? Or are rules for searching packages different than for modules? If so, are these rules covered somewhere in documentation?
UPDATE
I put it here for better formatting.
For futher investigation do the following:
- Remove root.pyc
- Rename root.py to root2.py
- Run python -vv -m unittest tests.test_1
# trying /home/dmugtasimov/tmp/name-res/root/root.so # trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so # trying /home/dmugtasimov/tmp/name-res/root/root.py # trying /home/dmugtasimov/tmp/name-res/root/root.pyc # trying /home/dmugtasimov/tmp/name-res/root/extra.so # trying /home/dmugtasimov/tmp/name-res/root/extramodule.so # trying /home/dmugtasimov/tmp/name-res/root/extra.py
It appears python disregard sys.path only for initial 4 tries.
UPDATE 2
Simplified version:
/home/dmugtasimov/tmp/name-res3/xyz
__init__.py
a.py
b.py
t.py
xyz.py
Files init.py, b.py and xyz.py are empty
File a.py:
import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
print 'sys.path is modified in a.py'
sys.path.insert(0, ROOT_DIRECTORY)
else:
print 'sys.path is NOT modified in a.py'
print 'sys.path:', sys.path
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'
File t.py:
import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
print 'sys.path is modified in t.py'
sys.path.insert(0, ROOT_DIRECTORY)
else:
print 'sys.path is NOT modified in t.py'
import xyz.a
Run:
python a.py
Output:
sys.path is modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b AFTER import xyz.b
Run:
python -vv a.py
Output:
import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc
Run:
python t.py
Output:
sys.path is modified in t.py sys.path is NOT modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b Traceback (most recent call last): File "t.py", line 9, in <module> import xyz.a File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module> import xyz.b ImportError: No module named b
Run:
python -vv t.py
Output:
import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/a.so # trying /home/dmugtasimov/tmp/name-res3/xyz/amodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/a.py # /home/dmugtasimov/tmp/name-res3/xyz/a.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/a.py import xyz.a # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/a.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/os.so # trying /home/dmugtasimov/tmp/name-res3/xyz/osmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/os.py # trying /home/dmugtasimov/tmp/name-res3/xyz/os.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.so # trying /home/dmugtasimov/tmp/name-res3/xyz/sysmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.py # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc # clear[2] __file__ # clear[2] __package__ # clear[2] sys # clear[2] ROOT_DIRECTORY # clear[2] __name__ # clear[2] os sys.path is modified in t.py sys.path is NOT modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b Traceback (most recent call last): File "t.py", line 9, in <module> import xyz.a File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module> import xyz.b ImportError: No module named b
As you see sys.path is the same for both cases:
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
But the behaviour is different. For a.py python searches for package xyz first, and them for module b in it:
import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc
In other words:
- Search PACKAGE xyz in directory sys.path[0] -> FOUND
- Search module b in PACKAGE xyz -> FOUND
- Continue execution
For t.py it searches for moduel xyz in the same directory as a.py itself and then fails to find module b in module xyz:
# trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc
In other words:
- Search MODULE xyz in directory in the same directory as a.py (or sys.path[1] ?) -> FOUND
- Search MODULE b in MODULE xyz -> NOT FOUND
- ImportError
So it looks like if "import xyz.b" bahaves different depending on how a.py was initially loaded as a script or imported from another module.
UPDATE 3
I filed proposal for documentation fix: http://bugs.python.org/issue16891
UPDATE 4
The reason for behavior described in UPDATE 2 is completely clear for me now.
http://docs.python.org/2/tutorial/modules.html#intra-package-references
6.4.2. Intra-package References
The submodules often need to refer to each other. For example, the surround module might use the echo module. In fact, such references are so common that the import statement first looks in the containing package before looking in the standard module search path. Thus, the surround module can simply use import echo or from echo import echofilter. If the imported module is not found in the current package (the package of which the current module is a submodule), the import statement looks for a top-level module with the given name.
For "python a.py" "a" is not considered as module in package, but for "python t.py" "a" is considered as a module in package "xyz". Therefore in first case it searches according sys.path, but in the second case it searches inside the same package (namely "xyz") for module named "xyz" (in other words "xyz.xyz")
You can easily see if change a.py like this:
File a.py:
import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
print 'sys.path is modified in a.py'
sys.path.insert(0, ROOT_DIRECTORY)
else:
print 'sys.path is NOT modified in a.py'
print 'sys.path:', sys.path
print '__package__', __package__
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'
My output was:
~/tmp/name-res3/xyz $ python a.py sys.path is modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] __package__ None BEFORE import xyz.b AFTER import xyz.b ~/tmp/name-res3/xyz $ python t.py sys.path is modified in t.py sys.path is NOT modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] __package__ xyz BEFORE import xyz.b Traceback (most recent call last): File "t.py", line 9, in <module> import xyz.a File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 12, in <module> import xyz.b ImportError: No module named b
Thanks to @J.F. Sebastian for pointing out the right place of documentation.
UPDATE 5
It seems that there is another issue. If interested, please, follow comments here: http://bugs.python.org/issue16891