Parallel Python: What is a callback?

Question

In Parallel Python it has something in the submit function called a callback (documentation) however it doesn't seem to explain it too well. I've posted on their forum a couple days ago and I've not received a response. Would someone explain what a callback is and what it's used for?

Thank you.

Answer 1

A callback is a function you define that's later called by a function you call.

As an example, consider how AJAX works: you write code that calls a back-end server function. At some point in the future, it returns from that function (the "A" stands for Asynchronous, which is what the "Parallel" in "Parallel Python" is all about). Now - because your code calls the code on the server, you want it to tell you when it's done, and you want to do something with its results. It does so by calling your callback function.

When the called function completes, the standard way for it to tell you it's done is for you to tell it to call a function in your code. That's the callback function, and its job is to handle the results/output from the lower-level function you've called.

Answer 2

The relevant spot in the docs:

callback - callback function which will be called with argument 
        list equal to callbackargs+(result,) 
        as soon as calculation is done
callbackargs - additional arguments for callback function

So, if you want some code to be executed as soon as the result is ready, you put that code into a function and pass that function as the callback argument. If you don't need other arguments, it will be just, e.g.:

def itsdone(result):
  print "Done! result=%r" % (result,)
...
submit(..., callback=itsdone)

For more on the callback pattern in Python, see e.g. my presentation here.

Answer 3

A callback is a function provided by the consumer of an API that the API can then turn around and invoke (calling you back). If I setup a Dr.'s appointment, I can give them my phone number, so they can call me the day before to confirm the appointment. A callback is like that, except instead of just being a phone number, it can be arbitrary instructions like "send me an email at this address, and also call my secretary and have her put it in my calendar.

Callbacks are often used in situations where an action is asynchronous. If you need to call a function, and immediately continue working, you can't sit there wait for its return value to let you know what happened, so you provide a callback. When the function is done completely its asynchronous work it will then invoke your callback with some predetermined arguments (usually some you supply, and some about the status and result of the asynchronous action you requested).

If the Dr. is out of the office, or they are still working on the schedule, rather than having me wait on hold until he gets back, which could be several hours, we hang up, and once the appointment has been scheduled, they call me.

In this specific case, Parallel Python's submit function will invoke your callback with any arguments you supply and the result of func, once func has finished executing.

Answer 4

Looking at the link, just looks like a hook which is called.

callback - callback function which will be called with argument list equal to callbackargs+(result,) as soon as calculation is done

The "as soon as calculation is done" bit seems ambiguous. The point, as far as I can see of this thing is that the submit() call distributes work to other servers and then returns. Because the finishing is asynchronous, rather block, it allows you to provide a function which is called when some unit of work finishes. If you do:

submit( ..., callback=work_finished, ... )

Then submit will ensure work_finished() is called when the unit of distributed work is completed on the target server.

When you call submit() you can provide a callback which is called in the same runtime as the caller of submit() ... and it is called after the distribution of the workload function is complete.

Kind of like "call foo(x,y) when you have done some stuff in submit()"

But yea, the documentation could be better. Have a ganders at the ppython source and see at which point the callback is called in submit()

Answer 5

A callback is simply a function. In Python, functions are just more objects, and so the name of a function can be used as a variable, like so:

def func():
    ...

something(func)

Note that many functions which accept a callback as an argument usually require that the callback accept certain arguments. In this case, the callback function will need to accept a list of arguments specified in callbackargs. I'm not familiar with Parallel Python so I don't know exactly what it wants.