combining tqdm with delayed execution with dask in

问题:

tqdm and dask are both amazing packages for iterations in python. While tqdm implements the needed progress bar, dask implements the multi-thread platform and they both can make iteration process less frustrating. Yet - I'm having troubles to combine them both together.

For example, the following code implements a delayed execution in dask, with tqdm.trange progress bar. The thing is that since the delayed is performed quickly, the progress bar ends immediately, while the real computation-time effort is done during the compute part.

from dask import delayed,compute
from tqdm import trange
from time import sleep

ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in trange(10):
    result.append(delayed(fun)(i))

print compute(result)

How can I attach the progress bar to the actual execution in compute command?

回答1:

Consider Dask's progress bar

from dask.diagnostics import ProgressBar

with ProgressBar():
    compute(result)

Build a diagnostic of your own

You can use this plugin architecture to get a signal at the end of every task. http://dask.pydata.org/en/latest/diagnostics.html

Here is an example of someone doing exactly this: https://github.com/tqdm/tqdm/issues/278