I have a large Visual Studio 2010 solution file (.sln
) that contains a large number of projects (~50
). Each project consists of roughly 20
.cpp
and .h
files. Building the entire project takes roughly 2 hours on a fast Intel i7 machine with all the dependencies cached locally on a fast SSD.
I am attempting to replicate an existing experiment I found on this topic, and need some clarification on how to:
- Get the parallelization I'm seeking to work in Visual Studio (with the GUI/IDE).
- Get these changes also working with automated command-line builds via
MSBuild.exe
.
First, are the /m
and /maxcpucount
options the same? I've read an article on the /m
option, which seems to build more projects in parallel. This appears to be the same option I set via the following actions in the GUI:
- Tools
- Options
- Projects and Solutions
- Build and Run
Maximum number of parallel project builds
There is another option, /MP
, which I can reach in the GUI via:
- Projects
- Properties
- Configuration Properties
- C/C++
- General
Multi-processor Compilation
Please correct me if I'm wrong on this, but this seems to suggest that the project will build multiple .cpp
files in parallel, but there's no option to specify how many, unless I have to manually set it in the text box (ie: /MP 4
or /MP4
.
It also seems that in order for /MP
to work, I need to disable minimal rebuild (ie: Enabled Minimal Rebuild: No (/GM-)
), and it also doesn't work with pre-compiled headers. I've worked around the pre-compiled headers issue by making the precompiled headers be a dedicated project that the solution builds first, before all other projects.
To the question: How can I implement these options for parallel project builds within the solution, and parallel file (.cpp
) builds within the projects, using MSBuild via the command-line, and confirm that they are working as expected? Are my assumptions above also correct? My end goal is to test building the project with up to the total number of cores being uses, or possibly even overloading it if my build process is I/O bound rather than CPU bound. I can do this easily enough in Linux via:
NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
NUMJOBS="$(echo "$NUMCPUS*1.5" | bc)"
NUMJOBS="$(printf '%.0f' $NUMJOBS)"
alias pmake='time nice make -j$NUMJOBS'
This allows me to, on my 8-core PC, queue up 12 build tasks at once, and make
automatically builds up to 12 .cpp
files in parallel, handling implicit build rules/dependencies, etc. I'm trying to achieve the same thing in Visual Studio Professional and MSBuild. When my builds are I/O bound, this approach has been the best at keeping all cores at ~100%
usage.
Thank you.
The
/MP
switch for the compiler (cl.exe
) and the/m
option formsbuild.exe
are indeed the way to go. Here are some relevant documentation extracts:So as you noticed this was all about the compiler, and no word about msbuild yet, since the compiler is a standalone tool. In other words: suppose you open 2 command windows and in both of them invoke a command like
cl /MP3 a.cpp b.cpp c.cpp
at the same time, you'd have 6 compiler processes running at the same time. Which is the behaviour you are after and which is also what msbuild can do.Output from invoking
msbuild /?
:Also more relevant information here, specifically the part where it says
Now if you'd just call
msbuild /m my.vcxproj
on a typical project this won't do anything special because there is only one project to build. However if inside an msbuild file there is an call to the MsBuild task likeand the /m option is passed to msbuild then msbuild will spawn multiple other msbuild processes per the rules laid out above. And coincidently that is exactly what happens when building a solution with msbuild. Msbuild first converts the solution to an actual msbuild file which calls the MsBuild task with a list of projects in the solution. As such multiple projects can be built in parallel and for that multiple msbuild processes are created. In turn, if these are C++ projects with the /MP option set, each of those projects will create multiple compiler processes yielding a maximum of
<max num cl processes> * <max num msbuild instances>
parallel compilations. It depends on your build machine what is the optimum, if you want you can play with the options (to do this efficiently each of your projects would import a common property sheet in which you set the /MP option else you'd have to edit all of them seperately) and look at the elapsed time msbuild reports when a build is done. I always just leave the arguments for /m and /MP to the default because I don't want other developpers using my projects to have possible suboptimal configurations, and because it easily maxes out CPU.You can achieve this on the command line by calling:
The number of resulting processes is going to be:
MSBuild =
PARALLEL_PROJECT_COUNT
Cl =PARALLEL_PROJECT_COUNT * PARALLEL_FILE_COUNT