How should I generate requirements.txt? Pip Freeze

Question

How should I generate requirements.txt for Python projects?

Here is the problem I am having with pip freeze. Suppose my package P requires A, B, C. Suppose C is a library that imports X, Y, Z, but only X is needed by P. Then if I:

1) Install A
2) Install B
3) Install C, which installs X, Y, Z
4) Do a pip freeze into P's requirements.txt 

Then P's requirements.txt will look like:

1) A
2) B
3) C
4) X
5) Y
6) Z

But Y and Z are not actually required in my Python installation for P to run.

As far as I can tell, running pip freeze to generate P's requirements will show you all dependencies of dependencies, and thus is a superset of P's actual dependencies.

Answer 1

The purpose of a virtualenv is to have total control over the packages installed.

Suppose you only listed A, B, C, and X. Every time you create a new virtualenv from that requirements file, you'll get the latest versions of Y and Z. There are several problems with this:

1. You can't know you're not using Y: For a sufficiently complex project, it is nearly impossible to audit every codepath to ensure C never calls into Y. You're not just worrying about your own code any more; you're worrying about C's code as well. This just doesn't scale.
2. Even if you're just importing Y, you're using it: Python allows arbitrary code execution at import time. A new version of Y could do all sorts of obnoxious things at import time, such as printing to stdout, monkey patching X, or just about anything else you can imagine. A well-designed Y shouldn't do these things, but you'll find the quality of packages on PyPI highly variable.
3. New versions of Y can pull in new dependencies: If you include a new version of Y, you could end up adding package W to your virtualenv too, because the new version of Y requires it. As more packages are added, the first two problems are exacerbated. Worse, you might find that the new version of Y depends on a newer version of X, in which case you won't end up with the packages you actually want.
4. Producing a known-good configuration is more important: pip freeze is not designed to figure out minimal requirements. It is designed to enable deploying a complete application to many different environments consistently. That means it will err on the side of caution and list everything which could reasonably affect your project.

For these reasons, you should not try to remove Y and Z from your requirements file.

Answer 2

There is a python module called pipreqs . It generates requirements.txt based on imports in the project.