I have ETA tasks that get sent to a Redis broker for Celery. It is a single celery and redis instance, both int he same machine.
The problem is, tasks are getting executed multiple times. I've seen tasks executed 4 to 11 times.
I set up the visibility timeout to be 12 hours, given that my ETA's are between 4-11 hours (determined at runtime):
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 12 * 60 * 60}
Even with that, tasks still get executed multiple times.
Initially, the task in question was not idempotent, so I tried adding in a DB check to make them idempotent.
it looks something like this:
@app.task
def foo(side_effect_action):
if side_effect_action.executed:
return ALREADY_EXECUTED
else:
do_side_effect()
side_effect_action.executed = True
side_effect_action.save() #hits the db
return JUST_EXECUTED
Turns out that the celery worker gets to the task before foo is able to call side_effect_action.save() and save the state, so in all cases when it's looking for side_effect_action.executed it is still False, and thus gets executed multiple times.
Any ideas how can I solve this issue?
