Make (Parallel Jobs) on Windows

Question

What setup works for GNU make parallel jobs (-j) on Windows?

I have tried setting the shell to cmd.exe using MinGW make 3.81, this works in creating the multiple processes but make fails with the "waiting for job" message.

Can this work and what is the best setup? (MinGW / Cygwin / ???) Can someone point me to a working example to test against?

Answer 1

As Far as I can understand, GNU Make's parallel jobs depend on the underlying platform to supply (CPU) load information. This is unfortunately done in a non-Windows-compatible way, and only the "make -j" command does anything scalingwise. That command (without max number of jobs) is potentially quite dangerous, as it will eat memory like a rabid starving dog (I tried it on Qt, and it crashed on QtWebkit, the largest project in Qt, 4GB RAM, Win7).

On second thought, I get the feeling make -j3 runs faster than make, but in light of what I can find on the web (forums, manual...) it seems very POSIX/linux specific functionality.

BTW, I'm surprised your question was voted down.

Answer 2

I've never had any promblems using make -jn under Cygwin. It works rather well. I regularly use it with Microsoft's cl.exe compiler. It just works out of the box for me. Very UNIX like, which is a Good Thing™. Build scripts port nicely to Linux and the Mac. Thoroughly recommended.

Never liked MinGW make since I got to a place where 5 back-slashes were too few, yet six was too many. Sigh! (To do with MinGW hacks to make backslash separated path names allowable in make AFAIK.)

Answer 3

I found this Danny Thorpe's blog entry that explains the problems with parallel make build on Windows. Basically what it suggests is to set the following environment variable first:

set SHELL=cmd.exe

With that, I was able to run the make -j style command, but unfortunately not the controlled make -jN style command.

Answer 4

A tip that might help CMake users

The -jN option does not work when a makefile calls make recursively as cmake-makefiles typically do. But there is a remedy that again works because in the generated makefiles CMake calls via such a syntax.

$(MAKE) -f CMakeFiles\Makefile2 <subproject>

This means you can modify the variable MAKE:

mingw32-make "MAKE=mingw32-make -j3"

Now every time a subproject-make is started, it again gets the "-j3" option. But note that this effectively does not limit the number of parallel compiles to 3 as you might expect. If you have more than 3 projects that don't depend on each other on the same hierarchy then all 3 projects will build parallel and each of them launches 3 compile steps. Resulting in 9 parallel compile steps.

When we take another close look into the top Makefile generated by cmake then we see that the target "all" essentially only starts a sub make.

$(MAKE) -f CMakeFiles\Makefile2 all

So we can remove one layer of subproject parallelism by calling

mingw32-make "MAKE=mingw32-make -j3" -f CMakeFiles\Makefile2 all

But this sacrifices the progress reporting.

I hope this helps nevertheless.

Answer 5

I've resolved this problem so I share the solution to everyone.

You can try MinGW-TDM (Home page: http://tdm-gcc.tdragon.net/) or MinGW-Equation (Download page: http://www.equation.com/servlet/equation.cmd?fa=fortran).

Install 32 bit version to D:\MinGW\MinGW32 for example (and 64 bit version to D:\MinGW\MinGW64 if you wish to use 64 bits compilation binaries)

Create a batch file as following then run it at your build directory:

Set MinGW_bin=D:\MinGW\MinGW32\bin
Set MSys_bin=D:\MinGW\msys\bin
Set Path=%MSys_bin%;%MinGW_bin%
mingw32-make SHELL=CMD.exe -j6 -f makefile

You can download MSYS from http://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS. It has many UNIX like utilities that required building many open source libraries, applications.

Where -j6 is the option to run 6 parallel compiling jobs.

TIP: you can set the option -jN where N = your numbers of CPU cores (threads) multiple by 1.5. In my case, my CPU has 4 threads so I set this value to 6.

Note: you can run

make.exe -jN -f makefile

without SHELL=CMD.exe option and your compilation is run parallel but it not compatible in some case (where make.exe is come from MSYS directory).

Regards,