Community
Participate
Working Groups
Created attachment 284632 [details] CPU Sample of build We're currently transitioning our project from PDE Build to Tycho and are pleasantly surprised by the performance of parallel builds. The initial resolving dependencies step is not parallel though and consumes a large chunk of the total build time. On my machine when building the project (~1800 modules + target platform) with mvn package -T 32, DefaultTychoResolver::resolveProject takes 360s (1/3 of the build). See the attached cpu sample. I had a cursory look at the code and it seems like resolveProject runs independent per project making the loop in TychoMavenLifecycleParticipant::afterProjectsRead trivial to parallelize (I'll submit a patch). Running resolveProjects also with 32 threads brings it down to 50s.
Created attachment 284633 [details] CPU sample with parallel resolveProject loop
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171626
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171655
@Julian: is the time actually spent at resolving bundles, or at downloading dependencies? Is your patch specifically intended to parallelize download of dependencies in afterProjectsRead? I'm working on some way to delay the download of dependencies (bug 567760) so that they're not fetched in this step (unless some packaging type like eclipse-plugin explicitly requests them early). The code actually has many locations where is expect dependencies to be already fetched and that are likely to turn into requests for fetching. So it looks like if the main goal is to make dependency download being parallel-able, similar change should happen at several locations.
(In reply to Mickael Istria from comment #4) > @Julian: is the time actually spent at resolving bundles, or at downloading > dependencies? I don't see any download in the sample. I assume all dependencies are already downloaded, as it's not the first build. The build is using a p2 <repository>, not a target definition. The majority of the time is spent in p2 resolution and another big chunk in resolveClasspath.
(In reply to Julian Honnen from comment #5) > I don't see any download in the sample. I assume all dependencies are > already downloaded, as it's not the first build. > The build is using a p2 <repository>, not a target definition. > The majority of the time is spent in p2 resolution and another big chunk in > resolveClasspath. OK good, so those 2 areas of work are not conflicting.
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171626 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=c77b6ceaf8f4bca8de791b685a8b776c0283ccaf
We usually try to keep "noteworthy" tickets opened until the notes are added to the N&N. Can you please add a comment about it to https://wiki.eclipse.org/Tycho/Release_Notes/2.2#New_and_Noteworthy ?
Already done, it's awaiting moderation.
(In reply to Julian Honnen from comment #9) > Already done, it's awaiting moderation. OK. Unfortunately, the wiki doesn't notify about changes to moderate. I did merge your change so your further edits to the wiki should happen automatically without moderation now.
I'm getting a ConcurrentModificationException in my build process after this change. Check it here: https://travis-ci.com/github/adisandro/MMINT/builds/198626260
(In reply to Alessio Di Sandro from comment #11) > I'm getting a ConcurrentModificationException in my build process after this > change. > Check it here: https://travis-ci.com/github/adisandro/MMINT/builds/198626260 Would you be able to submit a patch to avoid that?
The build uses a global LocalArtifactRepository (and LocalMetadataRepository) which is read and written during the resolving when downloading artifacts --> P2ResolverImpl::toResolutionResult I'll push a patch.
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171981
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171981 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=02c677be950b863fac9b922db6ceedaf55075930
