cmake finds wrong python libs

Question

I'm new to CMake and have trouble understanding some usage concepts.

I'm calling a python script from a c++ program:

#include <Python.h>
...
Py_Initialize();
PyRun_SimpleFile(...);
Py_Finalize();

The corresponding cmake entries in my cmake file are:

FIND_PACKAGE(PythonLibs REQUIRED)
...
TARGET_LINK_LIBRARIES(MyApplication ${PYTHON_LIBRARIES})

This works as long as my python script isn't using any modules installed into the site-packages directory, otherwise I get an ImportError. This question shows how to find the location of the site-packages directory with CMake, but what should I tell CMake to do with it?

EDIT: Problem solved. Turns out FIND_PACKAGE(PythonLibs) finds a different python installation from what I'm normally using (/usr/local/lib/libpython2.7.dylib instead of /Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib - I'm on mac), which is how I get standard python modules, but none that I installed myself. To change the PYTHONPATH back to normal, I added

try:
  import some_package
except ImportError:
  if "my_python_path" in sys.path: raise
  sys.path.append("my_python_path")

at the top of my python script.

Answer 1

You are effectively embedding python in your program when you do this. Did you call Py_Initialize() before PyRun_SimpleFile ? Have a look at Embedding Python in Another Application.

Py_Initialize() will set up sys.path and is required to set the python environment.

If you can find out where python is installed, it is possible to set python home to override python path calculations. Use Py_SetPythonHome() before Py_Initialize().

On posix type OSes, here is a comment in getpath.c (cpython implementation of path resolving):

/* Search in some common locations for the associated Python libraries.
 *
 * Two directories must be found, the platform independent directory
 * (prefix), containing the common .py and .pyc files, and the platform
 * dependent directory (exec_prefix), containing the shared library
 * modules.  Note that prefix and exec_prefix can be the same directory,
 * but for some installations, they are different.
 *
 * Py_GetPath() carries out separate searches for prefix and exec_prefix.
 * Each search tries a number of different locations until a ``landmark''
 * file or directory is found.  If no prefix or exec_prefix is found, a
 * warning message is issued and the preprocessor defined PREFIX and
 * EXEC_PREFIX are used (even though they will not work); python carries on
 * as best as is possible, but most imports will fail.
 *
 * Before any searches are done, the location of the executable is
 * determined.  If argv[0] has one or more slashes in it, it is used
 * unchanged.  Otherwise, it must have been invoked from the shell's path,
 * so we search $PATH for the named executable and use that.  If the
 * executable was not found on $PATH (or there was no $PATH environment
 * variable), the original argv[0] string is used.
 *
 * Next, the executable location is examined to see if it is a symbolic
 * link.  If so, the link is chased (correctly interpreting a relative
 * pathname if one is found) and the directory of the link target is used.
 *
 * Finally, argv0_path is set to the directory containing the executable
 * (i.e. the last component is stripped).
 *
 * With argv0_path in hand, we perform a number of steps.  The same steps
 * are performed for prefix and for exec_prefix, but with a different
 * landmark.
 *
 * Step 1. Are we running python out of the build directory?  This is
 * checked by looking for a different kind of landmark relative to
 * argv0_path.  For prefix, the landmark's path is derived from the VPATH
 * preprocessor variable (taking into account that its value is almost, but
 * not quite, what we need).  For exec_prefix, the landmark is
 * Modules/Setup.  If the landmark is found, we're done.
 *
 * For the remaining steps, the prefix landmark will always be
 * lib/python$VERSION/os.py and the exec_prefix will always be
 * lib/python$VERSION/lib-dynload, where $VERSION is Python's version
 * number as supplied by the Makefile.  Note that this means that no more
 * build directory checking is performed; if the first step did not find
 * the landmarks, the assumption is that python is running from an
 * installed setup.
 *
 * Step 2. See if the $PYTHONHOME environment variable points to the
 * installed location of the Python libraries.  If $PYTHONHOME is set, then
 * it points to prefix and exec_prefix.  $PYTHONHOME can be a single
 * directory, which is used for both, or the prefix and exec_prefix
 * directories separated by a colon.
 *
 * Step 3. Try to find prefix and exec_prefix relative to argv0_path,
 * backtracking up the path until it is exhausted.  This is the most common
 * step to succeed.  Note that if prefix and exec_prefix are different,
 * exec_prefix is more likely to be found; however if exec_prefix is a
 * subdirectory of prefix, both will be found.
 *
 * Step 4. Search the directories pointed to by the preprocessor variables
 * PREFIX and EXEC_PREFIX.  These are supplied by the Makefile but can be
 * passed in as options to the configure script.
 *
 * That's it!
 *
 * Well, almost.  Once we have determined prefix and exec_prefix, the
 * preprocessor variable PYTHONPATH is used to construct a path.  Each
 * relative path on PYTHONPATH is prefixed with prefix.  Then the directory
 * containing the shared library modules is appended.  The environment
 * variable $PYTHONPATH is inserted in front of it all.  Finally, the
 * prefix and exec_prefix globals are tweaked so they reflect the values
 * expected by other code, by stripping the "lib/python$VERSION/..." stuff
 * off.  If either points to the build directory, the globals are reset to
 * the corresponding preprocessor variables (so sys.prefix will reflect the
 * installation location, even though sys.path points into the build
 * directory).  This seems to make more sense given that currently the only
 * known use of sys.prefix and sys.exec_prefix is for the ILU installation
 * process to find the installed Python tree.
 */

Answer 2

The best way to solve the problem that the wrong version is found (for instance 3.0 instead of 2.7) is to specify the minimum version to find_package (this will choose any version >= 2.7):

FIND_PACKAGE(PythonLibs 2.7 REQUIRED)

or to get the exact version:

FIND_PACKAGE(PythonLibs 2.7.5 EXACT REQUIRED)

Answer 3

You can setup manually on cmake libs \usr\share\cmake-3.2.3\Modules\FindPythonLibs.cmake:

set(PYTHON_LIBRARY "\\usr\\lib\\python2.7")
set(PYTHON_INCLUDE_DIR "\\usr\\include\\python2.7")

Answer 4

You can tell cmake where to find this PythonLibs by specifying the path to your python libraries like this:

cmake -DPYTHON_LIBRARIES=/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib .

This will then set the ${PYTHON_LIBRARIES} inside cmake to the right path.

To find out which other possible options (besides PYTHON_LIBRARIES) you can give to cmake (with the -DARG option) try running

ccmake .

Then press c to configure, and t for advanced options.

For example, you might also want to set

-DPYTHON_LIBRARY='/softwarepath/Python/Python2.7/lib/libpython2.7.so'
-DPYTHON_INCLUDE='/softwarepath/Python/Python2.7/include'