I know that to allow make
to be multithreaded, I use the command make --jobs=X
where X
is usually equal to number of cores (or twice that or whatever).
I am debugging a makefile - actually consists of many makefiles - to work with the --jobs=X
option. Here's an example of why it currently doesn't:
T1:
mkdir D1
output_makefile.bat > ./D1/makefile
T2:
cd D1
make
Executing this with --jobs=X
will lead to a race condition because T1
is not specified as a dependency of T2
and eventually T2
will get built ahead of T1
; most of the bugs I need to fix are of this variety.
If X
in --jobs=X
is greater than the number of ?logical or physical? cores, the number of jobs executed simultaneously will be capped at the number of ?logical or physical? cores.
My machine has 4 physical/8 logical cores but the build machine that will be running our builds will have as many as 64 cores.
So I'm concerned that just because my makefile (a) builds the final output correctly (b) runs without errors on my machine with --jobs=4
does not mean it'll run correctly and without errors with --jobs=64
on a 64-core machine.
Is there a tool that will simulate make
executing in an environment that has more cores than the physical machine?
What about creating a virtual machine with 64 cores and run it on my 4-core machine; is that even allowed by VMPlayer?
UPDATE 1
I realized that my understanding of make
was incorrect: the number of job slots make
creates is equal to the --jobs=N
argument and not the number of cores or threads my PC has.
However, this by itself doesn't necessarily mean that make
will also execute those jobs in parallel even if I have fewer cores than jobs by using task-switching.
I need to confirm that ALL the jobs are being executed in parallel vs merely 'queued up' and waiting for the actively executing jobs to finish.
So I created a makefile with 16 targets - more than the num of threads or cores I have - and each recipe merely echos
the name of the target a configurable number of times.
make.mk
all: 1 2 3 4 ... 14 15 16
<target X>:
@loop_output.bat $@
loop_output.bat
@FOR /L %%G IN (1,1,2048) DO @echo (%1-%%G)
The output will be something like
(16-1) <-- Job 16
(6-1400)
(12-334)
(1-1616) <-- Job 1
(4-1661)
(15-113)
(11-632)
(2-1557)
(10-485)
(7-1234)
(5-1530)
The format is Job#X-Echo#Y
. The fact that I see (1-1616)
after (16-1)
means that make
is indeed executing target 16
at the same time as target 1
.
The alternative is that make
finishes jobs (1-#of cores/threads) and then takes another chunk of jobs equal to #num cores/threads but that's not what's happening.