How are Python modules (which are shared libraries

2019-04-24 09:38发布

问题:

I am running an interactive Python shell and trying to see the path from which a module is being imported using the 'inspect' module. The modules I am trying to import have Python wrappers around C++ APIs using SWIG.

The following snippet shows my steps :

>>> import os
>>> import inspect
>>>      
>>> import db
>>> inspect.getfile(db)
'mypath1/lib/db/db.pyc'
>>>
>>> import dart
>>> inspect.getfile(dart)
'mypath2/lib/dart.so'
>>>

My PYTHONPATH contains both mypath1/lib/db and mypath2/lib.

I was under the impression that in order to be able to load modules, the interpreter needs access to a .py file which then calls imp.load_module to load the required shared library (.so file). This is what I see in case of the db module which has a db.py file under mypath1/lib/db. However, dart does not have a .py file under mypath2/lib.

Is it possible to import a module without the .py file as is happening in the case of the dart module ?

回答1:

Python searches for several different files for any given import including a directory by that name and containing an __init__.py, an .so file for pure-native Python modules and .pyc files which can be used even if the .py is removed.

Run strace -o trace_output.txt python to see how this works. Partial example for import md5:

stat("/usr/lib/python2.7/md5", 0x7fff81ff16d0) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/md5.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/md5.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/md5module.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/md5.py", O_RDONLY) = 3

On my setup, it actually searches:

  • ~/.local/lib/python2.7/
  • /usr/local/lib/python2.7/dist-packages
  • /usr/lib/python2.7/dist-packages
  • /usr/lib/python2.7/

Within each directory, the pattern of calling stat to find a directory then looking for .so files, then .py is followed.

For more info on writing a purely native python module, see here: https://docs.python.org/2/extending/index.html