I have a python project that I have packaged locally as a wheel, and I want to install this wheel in a virtual environment in a docker container, based off python:alpine3.7. My problem is that sometimes i works, and sometimes it fails to resolve the dependencies in the wheel, seemingly at random.
The dependencies for the wheel is:
_install_requires = [
'numpy>=1.15.0',
'pandas>=0.21.0',
'scipy>=1.0.0',
'scikit-learn>=0.19.2'
]
And my dockerfile is as follows:
FROM python:alpine3.7
# install dependencies
# the lapack package is only in the community repository
RUN echo "http://dl-4.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk --update add --no-cache --virtual .build-deps \
gcc \
freetype-dev
RUN apk add --update lapack-dev python py-pip python-dev &&\
apk add --update linux-headers
# Install dependencies
RUN apk add --no-cache --virtual .build-deps \
gfortran \
musl-dev \
g++ \
make &&\
ln -s /usr/include/locale.h /usr/include/xlocale.h
RUN pip install --upgrade pip && pip install virtualenv
RUN virtualenv -p python3 ./venv
ADD dist/project-as-wheel-py3-none-any.whl .
RUN . ./venv/bin/activate && pip install ./project-as-wheel-py3-none-any.whl
When building the docker container, sometimes it fails, with a typical log being:
Processing /project-as-wheel-py3-none-any.whl
Collecting scikit-learn>=0.19.2 (from project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/25/b6/454cf208be93efa3db50ce06b732328c57ede005d1dcfa71d9a1548530b0/scikit-learn-0.19.2.tar.gz (9.7MB)
Collecting pandas>=0.21.0 (from project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/3e/56/82c4d4c049294f87ebd05b65fdcbc9ed68bd23fb0a7e4469caf9a75d199f/pandas-0.23.3.tar.gz (10.5MB)
Collecting numpy>=1.15.0 (from project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/3a/20/c81632328b1a4e1db65f45c0a1350a9c5341fd4bbb8ea66cdd98da56fe2e/numpy-1.15.0.zip (4.5MB)
Collecting scipy>=1.0.0 (from project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/07/76/7e844757b9f3bf5ab9f951ccd3e4a8eed91ab8720b0aac8c2adcc2fdae9f/scipy-1.1.0.tar.gz (15.6MB)
Collecting python-dateutil>=2.5.0 (from pandas>=0.21.0->project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
Collecting pytz>=2011k (from pandas>=0.21.0->project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB)
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas>=0.21.0->project-as-wheel==0.2.6)
Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: scikit-learn, pandas, numpy, scipy
Running setup.py bdist_wheel for scikit-learn: started
Running setup.py bdist_wheel for scikit-learn: finished with status 'error'
Complete output from command /venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-439jjmrz/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-ogmdp3dv --python-tag cp37:
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 168, in get_numpy_status
import numpy
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 148, in get_scipy_status
import scipy
ModuleNotFoundError: No module named 'scipy'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 269, in <module>
setup_package()
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 249, in setup_package
.format(numpy_req_str, instructions))
ImportError: Numerical Python (NumPy) is not installed.
scikit-learn requires NumPy >= 1.8.2.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
----------------------------------------
Failed building wheel for scikit-learn
Running setup.py clean for scikit-learn
Running setup.py bdist_wheel for pandas: started
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/8d/e7/58/7a882c73419e95e6709d925abb4b56c719c9be59ef826fe01c
Running setup.py bdist_wheel for numpy: started
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/77/d2/00/03180c22fc40b6f17f6a5c6ea0fa1689734a9cf4cf54179a1c
Running setup.py bdist_wheel for scipy: started
Running setup.py bdist_wheel for scipy: finished with status 'error'
Complete output from command /venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-439jjmrz/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-83ktidr8 --python-tag cp37:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-439jjmrz/scipy/setup.py", line 474, in <module>
setup_package()
File "/tmp/pip-install-439jjmrz/scipy/setup.py", line 450, in setup_package
from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'
----------------------------------------
Failed building wheel for scipy
Running setup.py clean for scipy
Complete output from command /venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-439jjmrz/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
`setup.py clean` is not supported, use one of the following instead:
- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
files that aren't checked into the git repo)
Add `--force` to your command to use it anyway if you must (unsupported).
----------------------------------------
Failed cleaning build dir for scipy
Successfully built pandas numpy
Failed to build scikit-learn scipy
Installing collected packages: scikit-learn, six, python-dateutil, pytz, numpy, pandas, scipy, project-as-wheel
Running setup.py install for scikit-learn: started
Running setup.py install for scikit-learn: finished with status 'error'
Complete output from command /venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-439jjmrz/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dwajrs1d/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python3.7/scikit-learn:
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 168, in get_numpy_status
import numpy
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 148, in get_scipy_status
import scipy
ModuleNotFoundError: No module named 'scipy'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 269, in <module>
setup_package()
File "/tmp/pip-install-439jjmrz/scikit-learn/setup.py", line 249, in setup_package
.format(numpy_req_str, instructions))
ImportError: Numerical Python (NumPy) is not installed.
scikit-learn requires NumPy >= 1.8.2.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
----------------------------------------
Command "/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-439jjmrz/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dwajrs1d/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python3.7/scikit-learn" failed with error code 1 in /tmp/pip-install-439jjmrz/scikit-learn/
What confuses me is that pip manages to install numpy and pandas, then fails to install scipy claiming there's "No module named 'numpy'".
Edit: A TL;DR on the log is:
- pip tries to install scikit-learn, fails as numpy is not installed.
- pip tries to install pandas, succeeds.
- pip tries to install numpy, succeeds.
- pip tries to install scipy, fail as numpy is not installed (however it was successfully installed in step 3 above!).
- pip declares that it succeeded in installing pandas and numpy, but failed to install scikit-learn and scipy.
- pip tries again to install scipy, fails again, complaining that numpy is not installed.
- pip gives up.
Edit2: Based on Nuts suggestion I changed the relevant line in the dockerfile to
RUN . ./venv/bin/activate && pip install numpy && pip install scipy && pip install ./project-as-wheel-py3-none-any.whl
Not very satisfactory, as I can't get the version requirements out of setup.py, but it works.
