pip packages not included in “conda env export”

Question

My python project runs on a conda virtual environment. I install packages on the environment with conda install whenever that package is available for installing that way, otherwise, I install it with pip install.

To make the project installation easier for other developers, I export the list of packages that are used on the conda environment to a file: conda env export > conda_environment_export.yml.

However, since a while, the package export does no longer contain the packages installed by pip.

What can I do to make pip packages appear again in the export?

My conda export file as it was some weeks ago:

name: sensor_gateway
channels:
  - defaults
dependencies:
  - aiohttp=2.3.9=py36_0
  - async-timeout=2.0.0=py36hc3e01a3_0
  - certifi=2018.1.18=py36_0
  - chardet=3.0.4=py36h420ce6e_1
  - jinja2=2.10=py36h292fed1_0
  - markupsafe=1.0=py36h0e26971_1
  - multidict=3.3.2=py36h72bac45_0
  - pip=9.0.1=py36h226ae91_4
  - pymysql=0.7.11=py36hf59f3ba_0
  - python=3.6.4=h6538335_1
  - pytz=2018.3=py36_0
  - setuptools=38.4.0=py36_0
  - sqlalchemy=1.2.1=py36hfa6e2cd_0
  - vc=14=h0510ff6_3
  - vs2015_runtime=14.0.25123=3
  - wheel=0.30.0=py36h6c3ec14_1
  - wincertstore=0.2=py36h7fe50ca_0
  - yarl=0.14.2=py36h27d1bf2_0
  - pip:
    - aiohttp-jinja2==0.16.0
    - aiomysql==0.0.12
    - attrs==17.4.0
    - idna==2.6
    - idna-ssl==1.0.0
    - python-mimeparse==1.6.0
prefix: C:\ProgramData\Anaconda3\envs\sensor_gateway

The conda export now:

name: sensor_gateway
channels:
  - defaults
dependencies:
  - aiohttp=2.3.9=py36_0
  - async-timeout=2.0.0=py36hc3e01a3_0
  - certifi=2018.4.16=py36_0
  - chardet=3.0.4=py36h420ce6e_1
  - icc_rt=2017.0.4=h97af966_0
  - intel-openmp=2018.0.0=8
  - jinja2=2.10=py36h292fed1_0
  - markupsafe=1.0=py36h0e26971_1
  - mkl=2018.0.2=1
  - mkl_fft=1.0.1=py36h452e1ab_0
  - mkl_random=1.0.1=py36h9258bd6_0
  - multidict=3.3.2=py36h72bac45_0
  - numpy=1.14.2=py36h5c71026_1
  - pip=9.0.1=py36h226ae91_4
  - pymysql=0.7.11=py36hf59f3ba_0
  - python=3.6.4=h6538335_1
  - pytz=2018.3=py36_0
  - setuptools=38.4.0=py36_0
  - simplejson=3.14.0=py36hfa6e2cd_0
  - sqlalchemy=1.2.1=py36hfa6e2cd_0
  - vc=14=h0510ff6_3
  - vs2015_runtime=14.0.25123=3
  - wheel=0.30.0=py36h6c3ec14_1
  - wincertstore=0.2=py36h7fe50ca_0
  - yarl=0.14.2=py36h27d1bf2_0
prefix: C:\ProgramData\Anaconda3\envs\sensor_gateway

I suspect that updating the version of conda (4.4.10-py36_0 --> 4.5.2-py36_0) may have caused this problem on this project. On another project, I have the same issue, but there I updated pip (9.0.1-py36h226ae91_4 --> 10.0.1-py36_0). Even after going back to pip 9.0.1 there, the problem remains. Is my suspicion correct? And what can I do to fix this again?

UPDATE:

The only proposed answer right now links to this question and it does not seem to solve my problem:

1. On a Windows command terminal, opened as administrator:

where pip output (where as windows-equivalent command to which on unix/linux):

C:\ProgramData\Anaconda3\Scripts\pip.exe
C:\Users\Sander\AppData\Local\Programs\Python\Python36\Scripts\pip.exe

pip freeze output:

Lists all packages I ever installed on my 'normal' python (= everything by default installed, not inside any dedicated virtual environments). The package list is too long to include here.

2. On a Windows command terminal, opened with administrator privileges, and after executing activate sensor_gateway:

where pip output:

C:\ProgramData\Anaconda3\envs\sensor_gateway\Scripts\pip.exe
C:\ProgramData\Anaconda3\Scripts\pip.exe
C:\Users\Sander\AppData\Local\Programs\Python\Python36\Scripts\pip.exe

pip freeze output:

lists all packages installed with pip on my environment correctly.

So it seems that when my virtual environment is activated, there is a pip installed in that environment, and it must be that pip that returns the list of installed packages when asked for, not another pip install, since the pip package list on the environment is correct.

Answer 1

I was able to solve this problem just by running this in a command window with admin privileges:

activate <env-name>
conda update pip

After that, exports of the available packages on my environment contain again both the packages installed with conda and those with pip.

I suspect I may have caused this package export problem because I had followed the instruction that pip gave earlier on, when I was executing a package install:

"You are using pip version 9.0.3, however version 10.0.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' > command."

So updating with python -m pip install --upgrade pip broke my package exports, but conda update pip fixed it.

Answer 2

At the moment of writing this post, because of an issue with reading pip lists by anaconda, conda export command only exports packages installed via conda install packagename and ignores the pip ones.

You can try reinstalling your packages via conda install and see if it generates the environment.yml file correctly.

Answer 3

For me this only appeared using the conda package pip 18.1_py36_0 (didnt try any other versions). So i downgraded pip in the virtual environment to version 10 via

conda install pip=10

After this the export was listing the pip packages again.

Also see this pull request for conda https://github.com/conda/conda/pull/7612 which fixes the issue for conda 4.6.2. It also was backported to conda 4.5.10. Currently there is no conda 4.4.x backport.

Answer 4

If you have installed pip with anaconda and used it to install the packages then conda-env does this automatically.

So basically you can export your environment with

conda env export -n <env-name> > environment.yml

Otherwise

Please refer to this link