Can't start Celery worker on Windows 10 with “

Question

I have a simple test code that runs successfully on Linux, but it won't run on my windows 10 x64 computer.

When I tried to start a celery worker, it complained about the unrecoverable error: PicklingError. (Celery version: 3.1.20)

In my celery config, I've set the serialization to 'json', but it still didn't help at all.

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

Here is the full error message:

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)   
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__
    self._create_worker_process(i)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process
    w.start()
  File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

Answer 1

I ran into the same problem. What was weird is that the problem only existed on Windows, on Linux Celery was running without any problems. Turns out I needed to pass the configuration module as name not as object:

app.config_from_object('celeryconfig')

instead of

app.config_from_object(celeryconfig)

Explanation from the Celery docs:

Tip

Using the name of a module is recommended as this means that the module doesn’t need to be serialized when the prefork pool is used. If you’re experiencing configuration pickle errors then please try using the name of a module instead.

Apparently Celery needs to pickle the configuration when it's passed as an object which fails on Windows. If passed as module name it works. Thanks for pointing me in the right direction @JoyLy!

Answer 2

An addition to @jeverling's answer.
If you'd like watch for a destination of your configs file, you can use __name__ variable.

from . import celery_config

celery_app.config_from_object(celery_config.__name__)