How to install and import Python modules at runtim

Question

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.

Here is the script.

#!/usr/bin/env python

import subprocess
import os
import sys

def pip_install(mod):
    print subprocess.check_output("pip install %s" % mod, shell=True)

if __name__ == "__main__":
    if os.getuid() != 0:
        print "Sorry, you need to run the script as root."
        sys.exit()

    try:
        import pexpect
    except:
        pip_install('pexpect') 
        import pexpect        

    # More code here...

The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.

How to install and import Python modules at runtime? I'm open to another approaches.

Answer 1

I actually made a module for this exact purpose (impstall)

It's really easy to use:

import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')

Github link for issues/contributions

Answer 2

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :

Updated answer of Paulo

import pip._internal as pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Answer 3

You can import pip instead of using subprocess:

import pip

def install(package):
    pip.main(['install', package])

# Example
if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Another take:

import pip

def import_with_auto_install(package):
    try:
        return __import__(package)
    except ImportError:
        pip.main(['install', package])
    return __import__(package)

# Example
if __name__ == '__main__':
    pexpect = import_with_auto_install('pexpect')
    print(pexpect)

[edit]

You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.

This is the output for me:

(test)root@vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
  Successfully uninstalled pexpect
(test)root@vagrant:~/test# python test.py
Downloading/unpacking pexpect
  Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
  Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
  Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root@vagrant:~/test#

Answer 4

I solved my problem using the imp module.

#!/usr/bin/env python

import pip
import imp

def install_and_load(package):
    pip.main(['install', package])

    path = '/usr/local/lib/python2.7/dist-packages'
    if path not in sys.path:
        sys.path.append(path)

    f, fname, desc = imp.find_module(package)
    return imp.load(package, f, fname, desc)

if __name__ == "__main__":
    try:
        import pexpect
    except:
        pexpect = install_and_load('pexpect')

    # More code...

Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.