Differences between distribute, distutils, setupto

Question

The Situation

I’m trying to port an open-source library to Python 3. (SymPy, if anyone is wondering.)

So, I need to run 2to3 automatically when building for Python 3. To do that, I need to use distribute. Therefore, I need to port the current system, which (according to the doctest) is distutils.

The Problem

Unfortunately, I’m not sure what’s the difference between these modules—distutils, distribute, setuptools. The documentation is sketchy as best, as they all seem to be a fork of one another, intended to be compatible in most circumstances (but actually, not all)…and so on, and so forth.

The Question

Could someone explain the differences? What am I supposed to use? What is the most modern solution? (As an aside, I’d also appreciate some guide on porting to Distribute, but that’s a tad beyond the scope of the question…)

Answer 1

This subject seems to still be in flux. As of 10-31-2013 the "Python Packaging User Guide" Quick Recommendations defines "what toolset is currently recommended". It also links to "The Future of Python Packaging"

Answer 2

I’m a distutils maintainer and distutils2/packaging contributor. I did a talk about Python packaging at ConFoo 2011 and these days I’m writing an extended version of it. It’s not published yet, so here are excerpts that should help define things.

• Distutils is the standard tool used for packaging. It works rather well for simple needs, but is limited and not trivial to extend.

• Setuptools is a project born from the desire to fill missing distutils functionality and explore new directions. In some subcommunities, it’s a de facto standard. It uses monkey-patching and magic that is frowned upon by Python core developers.

• Distribute is a fork of Setuptools that was started by developers feeling that its development pace was too slow and that it was not possible to evolve it. Its development was considerably slowed when distutils2 was started by the same group. 2013-August update: distribute is merged back into setuptools and discontinued.

• Distutils2 is a new distutils library, started as a fork of the distutils codebase, with good ideas taken from setup tools (of which some were thoroughly discussed in PEPs), and a basic installer inspired by pip. The actual name you use to import Distutils2 is packaging in the Python 3.3+ standard library, or distutils2 in 2.4+ and 3.1–3.2. (A backport will be available soon.) Distutils2 did not make the Python 3.3 release, and it was put on hold.

More info:

• The fate of Distutils – Pycon Summit + Packaging Sprint detailed report
• A Quick Diff between Distutils and Distutils2

I hope to finish my guide soon, it will contain more info about each library’s strong and weak points and a transition guide.

Answer 3

As of January 2017, all of the other answers to this question are at least two years out-of-date. When you come across advice on Python packaging issues, remember to look at the date of publication, and don't trust out-of-date information.

The Python Packaging User Guide is worth a read. Every page has a "last reviewed" date displayed, so you can check the recency of the manual, and it's quite comprehensive. The fact that it's hosted on a subdomain of python.org of the Python Software Foundation just adds credence to it. The Project Summaries page is especially relevant here.

Summary of tools:

Here's a summary of the Python packaging landscape in January 2017:

Supported tools:

• Distutils is still the standard tool for packaging in Python. It is included in the standard library (Python 2 and Python 3.0 to 3.6). It is useful for simple Python distributions, but lacks features. It introduces the distutils Python package that can be imported in your setup.py script.

  • _{Official docs | distutils section of Python Package User Guide}

• Setuptools was developed to overcome Distutils' limitations, and is not included in the standard library. It introduced a command-line utility called easy_install. It also introduced the setuptools Python package that can be imported in your setup.py script, and the pkg_resources Python package that can be imported in your code to locate data files installed with a distribution. One of its gotchas is that it monkey-patches the distutils Python package. It should work well with pip. It sees regular releases.

  • _{Official docs | Pypi page | GitHub repo | setuptools section of Python Package User Guide}

• scikit-build is an improved build system generator that internally uses CMake to build compiled Python extensions. Because scikit-build isn't based on distutils, it doesn't really have any of its limitations. When ninja-build is present, scikit-build can compile large projects over three times faster than the alternatives. It should work well with pip. It sees regular releases.

  • _{Official docs | Pypi page | GitHub repo}

Deprecated/abandoned tools:

• Distribute was a fork of Setuptools. It shared the same namespace, so if you had Distribute installed, import setuptools would actually import the package distributed with Distribute. Distribute was merged back into Setuptools 0.7, so you don't need to use Distribute any more. In fact, the version on Pypi is just a compatibility layer that installs Setuptools.

• Distutils2 was an attempt to take the best of Distutils, Setuptools and Distribute and become the standard tool included in Python's standard library. The idea was that Distutils2 would be distributed for old Python versions, and that Distutils2 would be renamed to packaging for Python 3.3, which would include it in its standard library. These plans did not go as intended, however, and currently, Distutils2 is an abandoned project. The latest release was in March 2012, and its Pypi home page has finally been updated to reflect its death.

Alpha software:

• Distlib is a tool that aims to implement a subset of the previous tools' functionality, but only functionality that is very well-defined in accepted PEPs. It is one of the tools of the PyPA (Python Package Authority), and it should hopefully be included eventually in the Python standard library someday. It is still considered alpha software, so end-users beware.

  • _{Official docs | Pypi page | Bitbucket repo | distlib section of Python Package User Guide}

• There are a couple more tools (eg: Bento), but I won't mention them as they are too obscure or niche or early or undeveloped for this answer post, or else they're not direct alternatives.

Recommendation:

So in conclusion, out of all these options, I would recommend Setuptools, unless your requirements are very basic and you only need Distutils. Setuptools works very well with Virtualenv and Pip, tools that I highly recommend. Virtualenv and Pip could both be considered official, as they're part of PyPA, and Python 3 now ships ensurepip (which helps you install pip on some systems).

If you're looking into Virtualenv, you might be interested in this question: What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, etc?. (Yes, I know, I groan with you.)

As a side-note, I recommend using Virtualenv 1.10 or higher, as it is the first release that recognises the Setuptools/Distribute merger, for both Python 2 and 3.

Answer 4

NOTE: Answer deprecated, Distribute now obsolete.

Yep, you got it. :-o I think at this time the preferred package is Distribute, which is a fork of setuptools, which are an extension of distutils (the original packaging system). Setuptools was not being maintained so is was forked and renamed, however when installed it uses the package name of setuptools! I think most Python developers now use Distribute, and I can say for sure that I do.

Answer 5

Updating this question in late 2014 where fortunately the Python packaging chaos has been greatly cleaned up by Continuum's "conda" package manager.

In particular, conda quickly enables the creation of conda "environments". You can configure your environments with different versions of Python. For example:

conda create -n py34 python=3.4 anaconda

conda create -n py26 python=2.6 anaconda

will create two ("py34" or "py26") Python environments with different versions of Python.

Afterwards you can invoke the environment with the specific version of Python with:

source activate <env name>

This feature seems especially useful in your case where you are having to deal with different version of Python.

Moreover, conda has the following features:

• Python agnostic
• Cross platform
• No admin privileges required
• Smart dependency management (by way of a SAT solver)
• Nicely deals with C, Fortran and system level libraries that you may have to link against

That last point is especially important if you are in the scientific computing arena.

Answer 6

I realize that I have replied to your secondary question without addressing unquestioned assumptions in your original problem:

I'm trying to port an open-source library (SymPy, if anyone is wondering) to Python 3. To do this, I need to run 2to3 automatically when building for Python 3.

You may, not need. Other strategies are described at http://docs.python.org/dev/howto/pyporting

To do that, I need to use distribute,

You may :) distutils supports build-time 2to3 conversion for code (not docstrings), in a different manner that distribute’s: http://docs.python.org/dev/howto/pyporting#during-installation