How to execute tasks in (FIFO) order using Google

Question

I believe task queues (push, pull, deferred) in Google App Engine do not guarantee that tasks will be executed in FIFO order. For example, suppose I have a task queue with tasks A, B, and C, and each task has timestamps t_A, t_B, and t_C, such that t_A < t_B < t_C. How can I ensure that tasks A, B, and C are executed in order of timestamp? If task B fails, I would like to delay the execution of task C until task B executes successfully. I've seen an ETA field to set the earliest time that a task can be sent, but this seems like more of a heuristic, not a guarantee.

Answer 1

Consider using the Pipelines API:

https://github.com/GoogleCloudPlatform/appengine-pipelines

They have already done the work for you:

https://github.com/GoogleCloudPlatform/appengine-pipelines/wiki/Python#user-content-execution-ordering

class LogWaitLogInOrder(pipeline.Pipeline):

  def run(self, message1, message2, delay):
    with pipeline.InOrder():
      yield LogMessage(message1)
      yield Delay(seconds=delay)
      yield LogMessage(message2)

    yield LogMessage('This would happen immediately on run')

The pipelines api also gives you reporting/feedback/notification/etc.

But if that's overkill for your particular needs, then just do as @Paul suggested, just create the next task when the first one completes and hope for the best

---

video reference:

Google I/O 2010 - Data pipelines with Google App Engine:

https://www.youtube.com/watch?v=zSDC_TU7rtc