Parallel Tasking Concurrency with Dependencies on

Question

I'm looking for a method or possibly a philosophical approach for how to do something like GNU Make within python. Currently, we utilize makefiles to execute processing because the makefiles are extremely good at parallel runs with changing single option: -j x. In addition, gnu make already has the dependency stacks built into it, so adding a secondary processor or the ability to process more threads just means updating that single option. I want that same power and flexibility in python, but I don't see it.

As an example:

all:  dependency_a dependency_b dependency_c

dependency_a:  dependency_d
    stuff

dependency_b:  dependency_d
    stuff

dependency_c:  dependency_e
    stuff

dependency_d:  dependency_f
    stuff

dependency_e:
    stuff

dependency_f:
    stuff

If we do a standard single thread operation (-j 1), the order of operation might be:

dependency_f -> dependency_d -> dependency_a -> dependency_b -> dependency_e \
             -> dependency_c

For two threads (-j 2), we might see:

1: dependency_f -> dependency_d -> dependency_a -> dependency_b

2: dependency_e -> dependency_c

Does anyone have any suggestions on either a package already built or an approach? I'm totally open, provided it's a pythonic solution/approach.

Please and Thanks in advance!

Answer 1

Have also a look at Waf, it's less complicated than Scons.

Waf is a Python-based framework for configuring, compiling and installing applications. Here are perhaps the most important features of Waf:

Automatic build order: the build order is computed from input and output files, among others Automatic dependencies: tasks to execute are detected by hashing files and commands Performance: tasks are executed in parallel automatically Flexibility: new commands can be added very easily through subclassing Features: support for lots of programming languages and compilers is included by default Documentation: the application is based on a robust model documented in The Waf book and in the API docs Python support: from Python 2.4 to 3.2 (Jython 2.5 and PyPy are also supported)

(from the website)

Answer 2

redo -j

"Smaller, easier, more powerful, and more reliable than make. An implementation of djb's redo" in Python.

Answer 3

You might want to have a look a jug. It's a task-based parallelisation framework that includes dependency tracking.

Answer 4

You should use Scons, as it already does the computations you want, and you can subvert it to do pretty much anything (like Make).

Answer 5

Have a look at Scons. It is a replacement for GNU Make written in Python.

SCons is a software construction tool—that is, a superior alternative to the classic "Make" build tool that we all know and love.

SCons is implemented as a Python script and set of modules, and SCons "configuration files" are actually executed as Python scripts. This gives SCons many powerful capabilities not found in other software build tools.