pip ignores dependency_links in setup.py

Question

I have dependency_links in my setup.py:

...
dependency_links = ['http://github.com/robot-republic/python-s3/tarball/master.tar.gz#egg=python-s3'],
...

But it doesn't work. However install_requires works fine. Maybe there are another method to set up git repo as required for setup.py?

Answer 1

I realize this is an old question, but, just in case you find yourself here like I did, this is what worked for me.

I've got a package on GitHub (not registered with pypi) that relies on other GitHub (non-pypi) packages. I spent an inordinate amount of time trying to figure out how to get pip to handle this correctly. I will include what I did to fix it here.

Putting dependencies in a requirements.txt file is the preferred method of listing dependencies. However, you also need to populate install_requires in setup. It was at this stage that I ran into a roadblock with pip not wanting to install dependencies from GitHub.

Most places, including answers to this question, tell you to populate the dependency_links section of setup. However, you also need to populate the install_requires field with the name of the package referenced in dependency_links.

For example, if your requirements.txt contains the following.

somepackage==1.2.0
https://github.com/user/repo/tarball/master#egg=repo-1.0.0
anotherpackage==4.2.1

Then, your setup call should look like this:

setup(
    name='yourpackage',
    version='1.7.5',
    packages=[],
    url='',
    license='',
    author='',
    author_email='',
    description='',
    install_requires=[
        'somepackage==1.2.0',
        'repo==1.0.0',
        'anotherpackage==4.2.1'
    ],
    dependency_links=[
        'https://github.com/user/repo/tarball/master#egg=repo-1.0.0'
    ]
)

Ok, so now we've got our package configured; installing it is the next task. This is where I spent a lot of time. I could not figure out why specifying dependency_links apparently did nothing. The trick is that in some cases, you need to set the allow-all-external (can be more specific) flag for pip. For example:

pip install git+https://github.com/user/anotherrepo.git
--process-dependency-links --allow-all-external

You're done and it works!

DISCLAIMER: dependency_links and the flags process-dependency-links and allow-all-external are deprecated, so they will be removed soon. In the time I spent, I could not locate a better, prefered method and still have pip function properly.

Answer 2

This answer should help. In a nutshell, you need to specify the version (or "dev") for the #egg=python-s3 so it looks like #egg=python-s3-1.0.0.

Updates based on @Cerin's comment:

• Pip 1.5.x has a flag to enable dependency-links processing: --process-dependency-links. I haven't tested it because I agree with the point below.
• This discussion seems to indicate that using dependency-links for pip is a bad practice. Although this feature was enlisted for deprecation, it's not anymore. There's a valid use case for private packages.

Answer 3

A couple of notes on some issues I found, in particular for installing from private repos.

Installing from pip & setuptools have some subtle differences; but this way should work for both.

from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

• For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
• You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
• You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
• version can be a branch, a tag, or a commit hash
• You need to supply --process-dependency-links if installing from pip