Installing the same python environment on another

Question

I'm developing a python project on my desktop (OS X) and it's running well. Now I need to run it on a computing cluster that is running Linux and I have no root access. On my Home in the computing cluster I installed Anaconda locally. Then when I run my project there I got so many strange bugs and errors. THe code runs fine on my desktop. So I thought probably the problem is that probably Anaconda has newr modules than what I have and that's probably causing errors. (backward compatibility issues)

Is there any way to use the same python modules versions I have in my desktop on the computing cluster?

I wonder if there is a command that produces all the packages and modules I have on my desktop with their versions and then import this file into some tool in that linux machine and then I get a cloned version of my setup. Is there something like that?

Answer 1

If you have pip installed you could try this pip freeze.

If you want to get list of modules from python shell:

help('modules')

Answer 2

Use pyenv to control the python version

pyenv is designed for installing python into the home folder so you don't need root access (but even with root access it's a great idea), it lets you use a specified version of python within your home folder hierarchy rather than the OS version. You should also install pyenv virtualenv so you can use a virtual environment for your project (not strictly essential, but virtual environments are a great idea and you should always use one, with pyenv they're practically effortless).

One of the neat things about pyenv is the pyenv local command which specifies which version of python (or which virtualenv) should be used for a folder (and subfolders), once you've used pyenv local in your project folder to set the python version any time you use the python command it'll use the version set by pyenv local. It's not needed if you only install one version of python, and don't use a virtualenv (in that case you can use pyenv global to set the version for the user). The neatest thing about pyenv local/global setup is it works great with both scripts and manually invoking python, it is simply set-and-forget unlike other python environment managers which require activation.

In brief, once you've set up pyenv you control exactly what version of python will run, and as it is installed into the home folder the OS has no power to affect it.

Having installed pyenv and pyenv virtualenv, you would then use it to install the same version of python as is used on your development machine, the commands you'd run would be something like this:

pyenv install 3.4.2
pyenv virtualenv 3.4.2 my_project_env
cd my_project
pyenv local my_project_env

Install modules into python environment

To get a list of python module version use pip freeze, you can do this on your development machine:

pip freeze > requirements.txt

Now copy requirements.txt to your deployment machines (with pyenv already setup using pyenv local or pyenv global) and run:

pip install -r requirements.txt

Which will install the same modules into the python environment.

Duplicating a pyenv

While it's kind of dodgy, once you've done this, you can even copy the entire installation (i.e. at least the .bashrc file and .pyenv folder) onto other machines, and if the machines have the same OS and the home folders have the same name, the transplanted environment should work fine. It might be more responsible to use a setup script but I have copied pyenv environments without anything terrible happening.