Expand Python Search Path to Other Source

Question

I have just joined a project with a rather large existing code base. We develop in linux and do not use and IDE. We run through the command line. I'm trying to figure out how to get python to search for the right path when I run project modules. For instance, when I run something like:

python someprojectfile.py

I get

ImportError: no module named core.'somemodule'

I get this for all of my imports to I assume it's an issue with the path.

TLDR:

How do I get Python to search ~/codez/project/ and all the files and folders for *.py files during import statements.

Answer 1

You should also read about python packages here: http://docs.python.org/tutorial/modules.html.

From your example, I would guess that you really have a package at ~/codez/project. The file __init__.py in a python directory maps a directory into a namespace. If your subdirectories all have an __init__.py file, then you only need to add the base directory to your PYTHONPATH. For example:

PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/project

In addition to testing your PYTHONPATH environment variable, as David explains, you can test it in python like this:

$ python
>>> import project                      # should work if PYTHONPATH set
>>> import sys
>>> for line in sys.path: print line    # print current python path

...

Answer 2

I read this question looking for an answer, and didn't like any of them.

So I wrote a quick and dirty solution. Just put this somewhere on your sys.path, and it'll add any directory under folder (from the current working directory), or under abspath:

#using.py

import sys, os.path

def all_from(folder='', abspath=None):
    """add all dirs under `folder` to sys.path if any .py files are found.
    Use an abspath if you'd rather do it that way.

    Uses the current working directory as the location of using.py. 
    Keep in mind that os.walk goes *all the way* down the directory tree.
    With that, try not to use this on something too close to '/'

    """
    add = set(sys.path)
    if abspath is None:
        cwd = os.path.abspath(os.path.curdir)
        abspath = os.path.join(cwd, folder)
    for root, dirs, files in os.walk(abspath):
        for f in files:
            if f[-3:] in '.py':
                add.add(root)
                break
    for i in add: sys.path.append(i)

>>> import using, sys, pprint
>>> using.all_from('py') #if in ~, /home/user/py/
>>> pprint.pprint(sys.path)
[
#that was easy
]

And I like it because I can have a folder for some random tools and not have them be a part of packages or anything, and still get access to some (or all) of them in a couple lines of code.

Answer 3

There are a few possible ways to do this:

• Set the environment variable PYTHONPATH to a colon-separated list of directories to search for imported modules.
• In your program, use sys.path.append('/path/to/search') to add the names of directories you want Python to search for imported modules. sys.path is just the list of directories Python searches every time it gets asked to import a module, and you can alter it as needed (although I wouldn't recommend removing any of the standard directories!). Any directories you put in the environment variable PYTHONPATH will be inserted into sys.path when Python starts up.
• Use site.addsitedir to add a directory to sys.path. The difference between this and just plain appending is that when you use addsitedir, it also looks for .pth files within that directory and uses them to possibly add additional directories to sys.path based on the contents of the files. See the documentation for more detail.

Which one of these you want to use depends on your situation. Remember that when you distribute your project to other users, they typically install it in such a manner that the Python code files will be automatically detected by Python's importer (i.e. packages are usually installed in the site-packages directory), so if you mess with sys.path in your code, that may be unnecessary and might even have adverse effects when that code runs on another computer. For development, I would venture a guess that setting PYTHONPATH is usually the best way to go.

However, when you're using something that just runs on your own computer (or when you have nonstandard setups, e.g. sometimes in web app frameworks), it's not entirely uncommon to do something like

from os.path import dirname
sys.path.append(dirname(__file__))

Answer 4

The easiest way I find is to create a file "any_name.pth" and put it in your folder "\Lib\site-packages". You should find that folder wherever python is installed.

In that file, put a list of directories where you want to keep modules for importing. For instance, make a line in that file like this:

C:\Users\example...\example

You will be able to tell it works by running this in python:

import sys
for line in sys: print line

You will see your directory printed out, amongst others from where you can also import. Now you can import a "mymodule.py" file that sits in that directory as easily as:

import mymodule

This will not import subfolders. For that you could imagine creating a python script to create a .pth file containing all sub folders of a folder you define. Have it run at startup perhaps.

Answer 5

I know this thread is a bit old, but it took me some time to get to the heart of this, so I wanted to share.

In my project, I had the main script in a parent directory, and, to differentiate the modules, I put all the supporting modules in a sub-folder called "modules". In my main script, I import these modules like this (for a module called report.py):

from modules.report import report, reportError

If I call my main script, this works. HOWEVER, I wanted to test each module by including a main() in each, and calling each directly, as:

python modules/report.py

Now Python complains that it can't find "a module called modules". The key here is that, by default, Python includes the folder of the script in its search path, BUT NOT THE CWD. So what this error says, really, is "I can't find a modules subfolder". The is because there is no "modules" subdirectory from the directory where the report.py module resides.

I find that the neatest solution to this is to append the CWD in Python search path by including this at the top:

import sys

sys.path.append(".")

Now Python searches the CWD (current directory), finds the "modules" sub-folder, and all is well.