Celery CRITICAL/MainProcess] Unrecoverable error:

Question

I've been running a flask application with a celery worker and redis in three separated docker containers without any issue.

This is how I start it:

celery worker -A app.controller.engine.celery -l info --concurrency=2 --pool eventlet

Celery starts fine:

 -------------- celery@a828bd5b0089 v4.2.1 (windowlicker)
---- **** -----
--- * ***  * -- Linux-4.9.93-linuxkit-aufs-x86_64-with 2018-11-15 16:06:59
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         app.controller.engine:0x7f8ba4eb70b8
- ** ---------- .> transport:   redis://redis:6379/0
- ** ---------- .> results:     redis://redis:6379/1
- *** --- * --- .> concurrency: 2 (eventlet)
-- ******* ---- .> task events: ON
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . app.controller.engine.do

INFO:engineio:Server initialized for eventlet.
INFO:engineio:Server initialized for threading.
[2018-11-15 15:44:34,301: INFO/MainProcess] Connected to redis://redis:6379/0
[2018-11-15 15:44:34,321: INFO/MainProcess] mingle: searching for neighbors
[2018-11-15 15:44:35,358: INFO/MainProcess] mingle: all alone
[2018-11-15 15:44:35,396: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.
[2018-11-15 15:44:35,415: INFO/MainProcess] celery@12af03844cd0 ready.

But when starting a task , it suddenly crashes:

[2018-11-15 15:02:03,224: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'float' object has no attribute 'items'",)
Traceback (most recent call last):
  File "/app/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
    c.loop(*c.loop_args())
  File "/app/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop
    connection.drain_events(timeout=2.0)
  File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
    get(self._deliver, timeout=timeout)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 359, in get
    ret = self.handle_event(fileno, event)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 341, in handle_event
    return self.on_readable(fileno), self
  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable
    chan.handlers[type]()
  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read
    self.connection._deliver(loads(bytes_to_str(item)), dest)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
    callback(message)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 632, in _callback
    self.qos.append(message, message.delivery_tag)
  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 149, in append
    pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
  File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd
    for pair in iteritems(mapping):
  File "/app/env/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems
    return iter(x.items())
AttributeError: 'float' object has no attribute 'items'

I don't see any reference to my code in the stacktrace.

This is the task method:

@celery.task()
def do(module_name, json_input):
    logger.info('____ Running _________________________')
    logger.info('testing ***********************')
    #modules.run(module_name, json_input)

I commented the call to my actual code just to verify I didn't mess it up with my modules code.. but it's crashing anyways.

Anybody has an idea what could be wrong or how to debug it properly?

Thanks a lot

Answer 1

The problem was that the my celery container downloaded a new release of the redis package with pip at build time, eventually with a bug or a serious change that crashes when celery tries to use it when connecting to redis. This new redis library is the 3.0.0 and it was released today.

To fix the issue you have to specify the previous version in the requirements.txt (which is 2.10.6).

And actually, specifying package versions in the requirements.txt is a good practice that prevents this type of problems.

Answer 2

I ran into this same issue today. hopefully this gets resolved shortly in an update by Celery. I reverted back to a previous version as well.

Answer 3

Not enough rep yet for a comment:

This issue is tracked by the celery folks in: https://github.com/celery/celery/issues/5175

Leading to this PR: https://github.com/celery/celery/pull/5176

Which has been merged and reverted. Because some other fix landed. I'm not sure if this fixed it. It is no longer relevant for me.

Answer 4

For anyone who use the docker-airflow, I used the folk of puckel/docker-airflow Just need to modify the dockerfile by adding

   && pip install 'redis>=2.10.5,<3'\

replace celery[redis] with only celery, by adding celery in apache-airflow built-in module i.e.,

  && pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql]==$AIRFLOW_VERSION 

Update: They also change the docker-airflow dockerfile recently to fix this bug, see here