Automatically setting jobs (-j) flag for a multico

Question

I have a Makefile on a machine that has a ton of cores in it, but I always seem to forget to write -jX when compiling my project and it takes way longer than it should.

Is there some way I can set the -j flag through an environment variable or some other persistent config file so that make will automatically execute multiple jobs in parallel on this machine?

Answer 1

At the beginning of a Makefile:

MAKEFLAGS+="j"

It won't take numeric arguments for any version of GNU Make before 4.2. After 4.2 you can do:

MAKEFLAGS+="j2"

For versions earlier than 4.2 if you happen to have some jobs running out of memory, you could make them run only one at a time with flock from util-linux-ng. For example, a convert utility from ImageMagick will use all resources it can get when used to optimize images according to Google's recommendations, so there's no point to have it run in parallel.

%.min.jpg: %.jpg
    @flock --wait 600 Makefile convert $< -sampling-factor 4:2:0 -strip $@

It is important to set a long wait time because make will still run most of these commands in parallel. Therefore, wait time must be as such as the deepest queue execution time. If you have eight cores, and eight large images take a minute to optimize, you must set the wait time to at least a minute.

With hundreds of cores and huge images, six hundred seconds set above might not be enough.

Answer 2

It appears that the MAKEFLAGS environment variable can pass flags that are part of every make run (at least for GNU make). I haven't had much luck with this myself, but it might be possible to use -l rather than -j to automatically run as many jobs as are appropriate for the number of cores you have available.

Answer 3

I'm assuming you're using Linux. This is from my ~/.bashrc

# parallel make
export NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
alias pmake='time nice make -j$NUMCPUS --load-average=$NUMCPUS'

sample usage

samm@host src> echo $NUMCPUS
8
samm@host src> pmake

becomes time nice make -j8 --load-average=8.

To answer your specific question about putting this into a Makefile, I don't find it practical to sprinkle this logic all over my Makefiles. Putting this into a top level Makefile also isn't a great solution since I often build from sub-directories and wish to build them in parallel as well. However, if you have a fairly flat source hierarchy, it may work for you.

Answer 4

Until sometime in 2016, you could put this in your makefile: (GNU make tested)

MAKEFLAGS += "-j$(NUM_CORES) -l$(NUM_CORES)

(where NUM_PPROCS is calculated or set according to one of many of the other answers here) And, bam! you have multi-process building going on.

Given that this has stopped working, the best thing that I could come up with is this, where the makefile calls itself, but with -jX and -lX.

ifeq ($(PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB),done)

all: ...
   ...

other_target: ...
    ...

else
# add parallelism equal to number of cores every time.
# "random" strings are to ensure uniqueness
NUM_CORES ?= $(shell grep -c "vendor_id" /proc/cpuinfo)
MAKEFLAGS +=" -j$(NUM_CORES) -l$(NUM_CORES) "

# for the default target case
parallel_wrapper_default_target_anthsqjkshbeohcbmeuthnoethoaeou:
    $(MAKE) PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

# catches everything else
% :
    $(MAKE) $@ PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

endif