Parallelization of recursive jobs in GNU make

Question

I am looking for an elegant way for the parallelization of jobs in GNU make. Here is a sample of what I did so far. Make processes the directories dir-1, dir-2 and dir-3 in a serial fashion which is logical but not my intention:

SUBDIRS=dir-1 dir-2 dir-3

default: all

all:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE)); done

.PHONY: clean

clean:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) clean); done

Is there a way to support parallel processing of these directories using the "-j" option without specifying specific targets for each directory?

Answer 1

This probably will not answer your question directly, but besides what the other answers suggest, I would recommend to look into non-recursive make techniques. That is truly an elegant way to parallelize build, although, depending on what are the existing Makefiles, can require significant effort. Non-recursive make has advantages not limited to easy parallelization: it sees a complete dependency graph and so it does not need to build too little or too much, meaning faster (sometimes much faster) build times.

Some resources:

• Discussion on SO
• The classic text about trouble with recursive make
• Design of non-recursive makefile
• Implementation
• Another implementation
• Benchmarks (quite outdated though)

Answer 2

Are dir-1, dir-2 and dir-3 interdependent or independent?

I have a similar structure but dependence between the subdirectories so with that I preferred to just use parallel builds within each of the subdirectories. You'd get that via

## default to four parallel runs
MAKEFLAGS += -j 4  

all:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) ); done

But another trick is to read up on SUBDIRS in the make manual -- you do not need the for loop as make can unroll this for you. Try something like

## default to four parallel runs
MAKEFLAGS += -j 4  

SUBDIRS =    dir-1 dir-2 dir-3

$(SUBDIRS):  #whatever your depends here
             $(MAKE) -C $@

Answer 3

I found that when using the :: (double-colon rule), I could not find a way to force order if there were any dependencies among app, lib, and doc. After reading much of the GNU Make manual, I came up with the following rules to enforce the dependencies at the same time as having a generic recursive make. Notice that the .PHONY rule is there to force make to enter the directory even though the directory already exists.

SUBDIRS = app lib doc

default: all

app: lib

.PHONY: $(SUBDIRS)

$(SUBDIRS):
    $(MAKE) -C $@ $(MAKECMDGOALS)

all clean install: $(SUBDIRS)

Answer 4

SUBDIRS = a b c

default: all

$(SUBDIRS)::
    $(MAKE) -C $@ $(MAKECMDGOALS)

all clean : $(SUBDIRS)