How to reduce the number of files in the anaconda

问题:

I'm running conda environments on a compute cluster where the total number of files per "project" is restricted (200k files max). I've only created a couple of conda environments (anaconda for Python 2.7; ~200 python & R packages installed in each environment; high package overlap between environments) and already hit that file number limit. Even when using conda clean -a only a small fraction of the files are removed. Some python packages in my conda environments (e.g., boost) contain >10k files, and clean does not reduce this.

Is there any way to greatly reduce the number of files stored as part of a conda environment?

回答1:

Anaconda uses hard links to reduce the consumed disk space. But if a limit is imposed on the number of files, each hard link counts.

As discussed in the comments, using Miniconda instead of Anaconda, and installing only the packages you actually need, might help.

If this isn't enough, I'd recommend to merge several of your environments into one. Then you'll have fewer hardlinks for the packages that overlap. Of course that is the opposite of what environments are there for, but such is the nature of workarounds.