Summary: | CDT Internal builder parallelization proposal | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Tools] CDT | Reporter: | Oleg Krasilnikov <oleg.krasilnikov> | ||||||||
Component: | cdt-build | Assignee: | Oleg Krasilnikov <oleg.krasilnikov> | ||||||||
Status: | RESOLVED FIXED | QA Contact: | |||||||||
Severity: | enhancement | ||||||||||
Priority: | P3 | CC: | bala.torati, oleg.krasilnikov | ||||||||
Version: | 4.0 | ||||||||||
Target Milestone: | 4.0 M5 | ||||||||||
Hardware: | All | ||||||||||
OS: | All | ||||||||||
Whiteboard: | |||||||||||
Attachments: |
|
Description
Oleg Krasilnikov
2006-09-11 09:54:38 EDT
Created attachment 49827 [details]
CDT Internal builder parallelization proposal
So this would be like a -j make flag for the internal builder. Very cool. This is also something some has asked me about with the indexer as well. Created attachment 51333 [details]
Patch which implements parallel
Parallel build mode is implemented for internal
builder, according to proposal in general.
Testing performed on x86-64 system with 2 Intel Xeon
processors with hyperthreading enabled. Both Win2003
and RH EL4 have reported 4 physical processors.
Note: number of CPUs detected automatically both for
Windows and Linux (not Mac yet) and used as "optimal
jobs number". But user can change threads number by
hand in Project Properties -> C/C++ Build -> Build
settings page. Maximal jobs number is not limited.
Below is a table representing absolute and relative time
for project build with different number of jobs(threads).
Standard code: 9.6 s 100%
1 thread 9.5 s 99%
2 threads 5.2 s 54%
4 threads 3.6 s 37.5%
8 threads 3.5 s 36.5%
16 threads 3.7 s 38.5%
We can see that maximum performance is achieved
if number of threads is 8 (not equal to CPU number !).
But it can differ for various kinds of projects.
On 1-processor system, performance does not depend
of jobs number (approximately the same build time
both for existing mode and parallel build with any
number of jobs).
Created attachment 51334 [details]
Patch which implements parallel
Parallel build mode is implemented for internal
builder, according to proposal in general.
Testing performed on x86-64 system with 2 Intel Xeon
processors with hyperthreading enabled. Both Win2003
and RH EL4 have reported 4 physical processors.
Note: number of CPUs detected automatically both for
Windows and Linux (not Mac yet) and used as "optimal
jobs number". But user can change threads number by
hand in Project Properties -> C/C++ Build -> Build
settings page. Maximal jobs number is not limited.
Below is a table representing absolute and relative time
for project build with different number of jobs(threads).
Standard code: 9.6 s 100%
1 thread 9.5 s 99%
2 threads 5.2 s 54%
4 threads 3.6 s 37.5%
8 threads 3.5 s 36.5%
16 threads 3.7 s 38.5%
We can see that maximum performance is achieved
if number of threads is 8 (not equal to CPU number !).
But it can differ for various kinds of projects.
On 1-processor system, performance does not depend
of jobs number (approximately the same build time
both for existing mode and parallel build with any
number of jobs).
Patch implemented. Code changes are made in HEAD. Oleg, this patch has caused a compile error. There are a number of people complaining, please follow the cdt-dev list. And please fix this as soon as possible. My apoplogies. Really, changes for "ProcessClosure.java" were not applied to head due to my miss. Currently, changes are made. Sorry once more. This seems not to work on my machine (CDT 4.0M5, P4 with hyperthreading => reported as 2 CPUs in Linux). I also cannot find something in the Preferences to specifiy the number of jobs manually. Is there something in the log files to look for? > This seems not to work on my machine (CDT 4.0M5, P4 with hyperthreading > => reported as 2 CPUs in Linux). Please explain how it behaves. > I also cannot find something in the Preferences > to specifiy the number of jobs manually. You are right. Currently we're creating separate job for each project, and it creates jobs for every configurstion involved. So number of resulting jobs cannot be set by user now, but it depends of selected projects and their configurations. Moreover, build process for each configuration may be split to several threads depending on settings in "Builder Settings" property page. So, real number of build threads is almost always bigger than number of physical processors (even HT). Moreover, most Java machines usually use extra processes to garbage collection and other utility jobs. In case of 2-processor system (moreover HT) we can be sure that they are loaded accordingly. > Is there something in the log files to look for? Not yet. Do you really need them ? |