Community
Participate
Working Groups
At the moment, all project resources are built sequentially in their dependancy order. Thus advantages of modern multi-core processors are not used effectively. On the other hand, preliminary tests have demonstrated perfomance boost almost proportionate to the number of processors (or processor cores) in case of parallel build. The idea is to build simultaneously all resources which do not depend on each other.
Created attachment 49827 [details] CDT Internal builder parallelization proposal
So this would be like a -j make flag for the internal builder. Very cool. This is also something some has asked me about with the indexer as well.
Created attachment 51333 [details] Patch which implements parallel Parallel build mode is implemented for internal builder, according to proposal in general. Testing performed on x86-64 system with 2 Intel Xeon processors with hyperthreading enabled. Both Win2003 and RH EL4 have reported 4 physical processors. Note: number of CPUs detected automatically both for Windows and Linux (not Mac yet) and used as "optimal jobs number". But user can change threads number by hand in Project Properties -> C/C++ Build -> Build settings page. Maximal jobs number is not limited. Below is a table representing absolute and relative time for project build with different number of jobs(threads). Standard code: 9.6 s 100% 1 thread 9.5 s 99% 2 threads 5.2 s 54% 4 threads 3.6 s 37.5% 8 threads 3.5 s 36.5% 16 threads 3.7 s 38.5% We can see that maximum performance is achieved if number of threads is 8 (not equal to CPU number !). But it can differ for various kinds of projects. On 1-processor system, performance does not depend of jobs number (approximately the same build time both for existing mode and parallel build with any number of jobs).
Created attachment 51334 [details] Patch which implements parallel Parallel build mode is implemented for internal builder, according to proposal in general. Testing performed on x86-64 system with 2 Intel Xeon processors with hyperthreading enabled. Both Win2003 and RH EL4 have reported 4 physical processors. Note: number of CPUs detected automatically both for Windows and Linux (not Mac yet) and used as "optimal jobs number". But user can change threads number by hand in Project Properties -> C/C++ Build -> Build settings page. Maximal jobs number is not limited. Below is a table representing absolute and relative time for project build with different number of jobs(threads). Standard code: 9.6 s 100% 1 thread 9.5 s 99% 2 threads 5.2 s 54% 4 threads 3.6 s 37.5% 8 threads 3.5 s 36.5% 16 threads 3.7 s 38.5% We can see that maximum performance is achieved if number of threads is 8 (not equal to CPU number !). But it can differ for various kinds of projects. On 1-processor system, performance does not depend of jobs number (approximately the same build time both for existing mode and parallel build with any number of jobs).
Patch implemented.
Code changes are made in HEAD.
Oleg, this patch has caused a compile error. There are a number of people complaining, please follow the cdt-dev list. And please fix this as soon as possible.
My apoplogies. Really, changes for "ProcessClosure.java" were not applied to head due to my miss. Currently, changes are made. Sorry once more.
This seems not to work on my machine (CDT 4.0M5, P4 with hyperthreading => reported as 2 CPUs in Linux). I also cannot find something in the Preferences to specifiy the number of jobs manually. Is there something in the log files to look for?
> This seems not to work on my machine (CDT 4.0M5, P4 with hyperthreading > => reported as 2 CPUs in Linux). Please explain how it behaves. > I also cannot find something in the Preferences > to specifiy the number of jobs manually. You are right. Currently we're creating separate job for each project, and it creates jobs for every configurstion involved. So number of resulting jobs cannot be set by user now, but it depends of selected projects and their configurations. Moreover, build process for each configuration may be split to several threads depending on settings in "Builder Settings" property page. So, real number of build threads is almost always bigger than number of physical processors (even HT). Moreover, most Java machines usually use extra processes to garbage collection and other utility jobs. In case of 2-processor system (moreover HT) we can be sure that they are loaded accordingly. > Is there something in the log files to look for? Not yet. Do you really need them ?
