celery 'Worker-n' pid:xxxx exited with 

问题:

In my tasks.py file, when I import hmmlearn,

from hmmlearn import hmm

and start my celery workers, I get the following error

[2017-06-14 09:18:27,638: INFO/MainProcess] Received task: 
sm.tasks.mytask[4e46806e-6f0f-420f-baac-c727c2a382d4]
[2017-06-14 09:18:27,716: ERROR/MainProcess] Process 'Worker-4' pid:5264 
exited with 'exitcode 1'
[2017-06-14 09:18:29,857: ERROR/MainProcess] Process 'Worker-7' pid:3172 
exited with 'exitcode 1'
[2017-06-14 09:18:29,857: ERROR/MainProcess] Process 'Worker-6' pid:5768 
exited with 'exitcode 1'
[2017-06-14 09:18:29,857: ERROR/MainProcess] Process 'Worker-5' pid:5236 
exited with 'exitcode 1'
[2017-06-14 09:18:31,450: ERROR/MainProcess] Process 'Worker-8' pid:5876 
exited with 'exitcode 1'

And after I shutdown the worker,

[2017-06-14 09:19:28,545: WARNING/MainProcess] c:\anaconda3\lib\site-
packages\celery\apps\worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

If I just comment out that import and code using that import, everything works fine. But, I'm able to execute all the tasks(including the hmm code) as standalone python code on ipython without any issues.

I'm using the conda distribution with following details

Current conda install:

           platform : win-64
      conda version : 4.3.21
   conda is private : False
  conda-env version : 4.3.21
conda-build version : 1.21.3
     python version : 3.5.2.final.0
   requests version : 2.14.2

λ conda list | grep celery
celery                    3.1.18                    <pip>

λ conda list | grep kombu
kombu                     3.0.37                    <pip>

λ conda list | grep hmmlearn
hmmlearn                  0.1.1               np111py35_0    omnia

What should I do?

回答1:

This might be because celery 3.1.xx comes bundled with billiard 3.3.

If you upgrade that package (to 3.5 at time of writing), the service might work again.

pip install --upgrade billiard

回答2:

I just stumbled upon a similar situation. Upgrading billiard to 3.5, as suggested in a different answer, was not an option (because it conflicts with Celery==3.1.25 and I prefer that particular version for Windows its support).

I figured out, however, that in my case the problem was most probably due to this issue - it occurred only when I tried to import anything from sklearn in the Worker's process.

The problem was resolved by adding this snippet before the imports from sklearn:

from multiprocessing import current_process
try:
    current_process()._config
except AttributeError:
    current_process()._config = {'semprefix': '/mp'}