How can I get a list of locally installed Python m

Question

I would like to get a list of Python modules, which are in my Python installation (UNIX server).

How can you get a list of Python modules installed in your computer?

Answer 1

Now, these methods I tried myself, and I got exactly what was advertised: All the modules.

Alas, really you don't care much about the stdlib, you know what you get with a python install.

Really, I want the stuff that I installed.

What actually, surprisingly, worked just fine was:

pip freeze

Which returned:

Fabric==0.9.3
apache-libcloud==0.4.0
bzr==2.3b4
distribute==0.6.14
docutils==0.7
greenlet==0.3.1
ipython==0.10.1
iterpipes==0.4
libxml2-python==2.6.21

I say "surprisingly" because the package install tool is the exact place one would expect to find this functionality, although not under the name 'freeze' but python packaging is so weird, that I am flabbergasted that this tool makes sense. Pip 0.8.2, Python 2.7.

Answer 2

This solution is primary based on modules importlib and pkgutil and work with CPython 3.4 and CPython 3.5, but has no support for the CPython 2.

---

Explanation

1. sys.builtin_module_names - names all built-in modules (look my answer here)
2. pkgutil.iter_modules() - returns an information about all available modules
3. importlib.util.find_spec() - returns an information about importing module, if exists
4. BuiltinImporter - an importer for built-in modules (docs)
5. SourceFileLoader - an importer for a standard Python module (by default has extension *.py) (docs)
6. ExtensionFileLoader - an importer for modules as shared library (written on the C or C++)

---

Full code

import sys
import os
import shutil
import pkgutil
import importlib
import collections

if sys.version_info.major == 2:
    raise NotImplementedError('CPython 2 is not supported yet')


def main():

    # name this file (module)
    this_module_name = os.path.basename(__file__).rsplit('.')[0]

    # dict for loaders with their modules
    loaders = collections.OrderedDict()

    # names`s of build-in modules
    for module_name in sys.builtin_module_names:

        # find an information about a module by name
        module = importlib.util.find_spec(module_name)

        # add a key about a loader in the dict, if not exists yet
        if module.loader not in loaders:
            loaders[module.loader] = []

        # add a name and a location about imported module in the dict
        loaders[module.loader].append((module.name, module.origin))

    # all available non-build-in modules
    for module_name in pkgutil.iter_modules():

        # ignore this module
        if this_module_name == module_name[1]:
            continue

        # find an information about a module by name
        module = importlib.util.find_spec(module_name[1])

        # add a key about a loader in the dict, if not exists yet
        loader = type(module.loader)
        if loader not in loaders:
            loaders[loader] = []

        # add a name and a location about imported module in the dict
        loaders[loader].append((module.name, module.origin))

    # pretty print
    line = '-' * shutil.get_terminal_size().columns
    for loader, modules in loaders.items():
        print('{0}\n{1}: {2}\n{0}'.format(line, len(modules), loader))
        for module in modules:
            print('{0:30} | {1}'.format(module[0], module[1]))


if __name__ == '__main__':
    main()

---

Usage

For the CPython3.5 (truncated)

$ python3.5 python_modules_info.py 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
30: <class '_frozen_importlib.BuiltinImporter'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
_ast                           | built-in
_codecs                        | built-in
_collections                   | built-in
_functools                     | built-in
_imp                           | None
_io                            | built-in
_locale                        | built-in
_operator                      | built-in
_signal                        | built-in
_sre                           | built-in
_stat                          | built-in
_string                        | built-in
_symtable                      | built-in
_thread                        | built-in
(****************************truncated*******************************)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
227: <class '_frozen_importlib_external.SourceFileLoader'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__future__                     | /usr/local/lib/python3.5/__future__.py
_bootlocale                    | /usr/local/lib/python3.5/_bootlocale.py
_collections_abc               | /usr/local/lib/python3.5/_collections_abc.py
_compat_pickle                 | /usr/local/lib/python3.5/_compat_pickle.py
_compression                   | /usr/local/lib/python3.5/_compression.py
_dummy_thread                  | /usr/local/lib/python3.5/_dummy_thread.py
_markupbase                    | /usr/local/lib/python3.5/_markupbase.py
_osx_support                   | /usr/local/lib/python3.5/_osx_support.py
_pydecimal                     | /usr/local/lib/python3.5/_pydecimal.py
_pyio                          | /usr/local/lib/python3.5/_pyio.py
_sitebuiltins                  | /usr/local/lib/python3.5/_sitebuiltins.py
(****************************truncated*******************************)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
64: <class '_frozen_importlib_external.ExtensionFileLoader'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
_bisect                        | /usr/local/lib/python3.5/lib-dynload/_bisect.cpython-35m-x86_64-linux-gnu.so
_bz2                           | /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
_codecs_cn                     | /usr/local/lib/python3.5/lib-dynload/_codecs_cn.cpython-35m-x86_64-linux-gnu.so
_codecs_hk                     | /usr/local/lib/python3.5/lib-dynload/_codecs_hk.cpython-35m-x86_64-linux-gnu.so
_codecs_iso2022                | /usr/local/lib/python3.5/lib-dynload/_codecs_iso2022.cpython-35m-x86_64-linux-gnu.so
(****************************truncated*******************************)

For the CPython3.4 (truncated)

$ python3.4 python_modules_info.py
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
54: <class '_frozen_importlib.BuiltinImporter'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
_ast                           | built-in
_bisect                        | built-in
_codecs                        | built-in
_collections                   | built-in
_datetime                      | built-in
_elementtree                   | built-in
_functools                     | built-in
_heapq                         | built-in
_imp                           | None
_io                            | built-in
_locale                        | built-in
_md5                           | built-in
_operator                      | built-in
_pickle                        | built-in
_posixsubprocess               | built-in
_random                        | built-in
(****************************truncated*******************************)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
246: <class '_frozen_importlib.SourceFileLoader'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__future__                     | /usr/lib/python3.4/__future__.py
_bootlocale                    | /usr/lib/python3.4/_bootlocale.py
_collections_abc               | /usr/lib/python3.4/_collections_abc.py
_compat_pickle                 | /usr/lib/python3.4/_compat_pickle.py
_dummy_thread                  | /usr/lib/python3.4/_dummy_thread.py
_markupbase                    | /usr/lib/python3.4/_markupbase.py
_osx_support                   | /usr/lib/python3.4/_osx_support.py
_pyio                          | /usr/lib/python3.4/_pyio.py
(****************************truncated*******************************)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
44: <class '_frozen_importlib.ExtensionFileLoader'>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
_bz2                           | /usr/lib/python3.4/lib-dynload/_bz2.cpython-34m-x86_64-linux-gnu.so
_codecs_cn                     | /usr/lib/python3.4/lib-dynload/_codecs_cn.cpython-34m-x86_64-linux-gnu.so
_codecs_hk                     | /usr/lib/python3.4/lib-dynload/_codecs_hk.cpython-34m-x86_64-linux-gnu.so
_codecs_iso2022                | /usr/lib/python3.4/lib-dynload/_codecs_iso2022.cpython-34m-x86_64-linux-gnu.so
_codecs_jp                     | /usr/lib/python3.4/lib-dynload/_codecs_jp.cpython-34m-x86_64-linux-gnu.so
_codecs_kr                     | /usr/lib/python3.4/lib-dynload/_codecs_kr.cpython-34m-x86_64-linux-gnu.so
_codecs_tw                     | /usr/lib/python3.4/lib-dynload/_codecs_tw.cpython-34m-x86_64-linux-gnu.so
_crypt                         | /usr/lib/python3.4/lib-dynload/_crypt.cpython-34m-x86_64-linux-gnu.so
(****************************truncated*******************************)

Answer 3

pip freeze does it all finding packages however one can simply write the following command to list all paths where python packages are.

>>> import site; site.getsitepackages()
['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

Answer 4

In normal shell just use

pydoc modules

Answer 5

Solution

My 50 cents for getting a pip freeze-like list from a Python script:

import pip
installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version)
     for i in installed_packages])
print(installed_packages_list)

As a (too long) one liner:

sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])

Giving:

['behave==1.2.4', 'enum34==1.0', 'flask==0.10.1', 'itsdangerous==0.24', 
 'jinja2==2.7.2', 'jsonschema==2.3.0', 'markupsafe==0.23', 'nose==1.3.3', 
 'parse-type==0.3.4', 'parse==1.6.4', 'prettytable==0.7.2', 'requests==2.3.0',
 'six==1.6.1', 'vioozer-metadata==0.1', 'vioozer-users-server==0.1', 
 'werkzeug==0.9.4']

Scope

This solution applies to the system scope or to a virtual environment scope, and covers packages installed by setuptools, pip and (god forbid) easy_install.

My use case

I added the result of this call to my flask server, so when I call it with http://example.com/exampleServer/environment I get the list of packages installed on the server's virtualenv. It makes debugging a whole lot easier.

Caveats

I have noticed a strange behaviour of this technique - when the Python interpreter is invoked in the same directory as a setup.py file, it does not list the package installed by setup.py.

Steps to reproduce:

Create a virtual environment

$ cd /tmp
$ virtualenv test_env
New python executable in test_env/bin/python
Installing setuptools, pip...done.
$ source test_env/bin/activate
(test_env) $ 

Clone a git repo with setup.py

(test_env) $ git clone https://github.com/behave/behave.git
Cloning into 'behave'...
remote: Reusing existing pack: 4350, done.
remote: Total 4350 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (4350/4350), 1.85 MiB | 418.00 KiB/s, done.
Resolving deltas: 100% (2388/2388), done.
Checking connectivity... done.

We have behave's setup.py in /tmp/behave:

(test_env) $ ls /tmp/behave/setup.py
/tmp/behave/setup.py

Install the python package from the git repo

(test_env) $ cd /tmp/behave && python setup.py install
running install
...
Installed /private/tmp/test_env/lib/python2.7/site-packages/enum34-1.0-py2.7.egg
Finished processing dependencies for behave==1.2.5a1

If we run the aforementioned solution from /tmp

>>> import pip
>>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['behave==1.2.5a1', 'enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']
>>> import os
>>> os.getcwd()
'/private/tmp'

If we run the aforementioned solution from /tmp/behave

>>> import pip
>>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']
>>> import os
>>> os.getcwd()
'/private/tmp/behave'

behave==1.2.5a1 is missing from the second example, because the working directory contains behave's setup.py file.

I could not find any reference to this issue in the documentation. Perhaps I shall open a bug for it.

Answer 6

help('modules')

in a Python shell/prompt.