Bug 569617 - speed up TychoTargetMapping::getPrimaryArtifact
Summary: speed up TychoTargetMapping::getPrimaryArtifact
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Tycho (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 10
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks:
 
Reported: 2020-12-10 05:05 EST by Julian Honnen CLA
Modified: 2021-04-28 16:52 EDT (History)
2 users (show)

See Also:

Attachments
CPU sample (20.51 KB, application/octet-stream)
2020-12-10 05:05 EST, Julian Honnen CLA 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Honnen CLA 2020-12-10 05:05:06 EST 
Created attachment 285013 [details]
CPU sample

The scanning for projects... step spends a significant portion of its time in in the listFiles call of TychoTargetMapping::getPrimaryArtifact, caused by the File.isFile() check.

public boolean accept(File file) {
	return file.isFile() && file.getName().endsWith(TARGET_EXTENSION)
			&& !file.getName().startsWith(".polyglot.");
}

An easy improvement should be to check the file name before doing file IO.
Comment 1 Eclipse Genie CLA 2020-12-10 05:13:12 EST 
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/173647
Comment 2 Mickael Istria CLA 2020-12-10 05:22:54 EST 
That's a good analysis and improvement! Did you verify that you actually save those ~20 seconds after the patch is applied?
Comment 3 Julian Honnen CLA 2020-12-10 06:20:10 EST 
(In reply to Mickael Istria from comment #2)
> That's a good analysis and improvement! Did you verify that you actually
> save those ~20 seconds after the patch is applied?
Not at that time and indeed the major performance issue was elsewhere.

We have an additional custom pomless extension modeled after TychoAggregatorMapping, to support our legacy project structure.
Disabling the original TychoAggregatorMapping (via -Dtycho.pomless.aggregator.names=-) actually saved minutes.

After that the savings of this change were negligible. I'll leave the decision to you.
Comment 4 Christoph Laeubrich CLA 2020-12-10 06:54:09 EST 
Pomless is always computed at runtime and thus of course has performance overhead (especially the aggregator part as it requires to perform several guess-work).
You can use a static pom.tycho for the aggregator (that should also support different project strcutures).
Comment 5 Julian Honnen CLA 2020-12-10 07:00:26 EST 
(In reply to Christoph Laeubrich from comment #4)
> Pomless is always computed at runtime and thus of course has performance
> overhead (especially the aggregator part as it requires to perform several
> guess-work).
> You can use a static pom.tycho for the aggregator (that should also support
> different project strcutures).
Our projects looks like this:
- root
  - A
    - plugins
    - features
    - testplugins
  - B
    - plugins
    - features
    - testplugins
  ...

Each containing 100s of modules, so the static pom.tycho was not an option ;)

The tycho aggregator doesn't support multiple "plugins" directories within a reactor and our custom one does also optionally skip the testplugins, speeding up the dependency resolution when we want to compile without tests.