Bug 335153 - Regression: p2 downloads are much slower due to picking remote artifacts even when a local file: URL is available
Summary: Regression: p2 downloads are much slower due to picking remote artifacts even...
Status: RESOLVED FIXED
Alias: None
Product: Equinox
Classification: Eclipse Project
Component: p2 (show other bugs)
Version: 3.6.1   Edit
Hardware: PC All
: P3 critical (vote)
Target Milestone: 3.7 M6   Edit
Assignee: DJ Houghton CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks: 335610 408208
  Show dependency tree
 
Reported: 2011-01-24 03:02 EST by Meng Xin Zhu CLA
Modified: 2013-11-10 22:32 EST (History)
5 users (show)

See Also:


Attachments
test case and proposed fix (11.53 KB, patch)
2011-01-25 04:27 EST, Meng Xin Zhu CLA
pascal: iplog+
Details | Diff
test data (7.04 KB, application/octet-stream)
2011-01-25 04:30 EST, Meng Xin Zhu CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Meng Xin Zhu CLA 2011-01-24 03:02:27 EST
ProvisioningContext itself has a private method named 'getLoadedArtifactRepositories' that is sorted all the available repositories, the local repository has higher priority.

There is no public API to get all artifact repositories from ProvisioningContext, so clients use below query to query out all the artifact repositories from ProvisioningContext instance like DownloadManager. The query result is stored by a set container that never keeps the results in a certain sequence.

If some artifact keys with same id and version in both local repository and remote repository, DownloadManager probably downloads them from remote repository. It's very annoying to cost much more time and bandwidth to download them.

The code snippet of DownloadManager.start():

IQueryable<IArtifactRepository> repoQueryable = provContext.getArtifactRepositories(subMonitor.newChild(250));
IQuery<IArtifactRepository> all = new ExpressionMatchQuery<IArtifactRepository>(IArtifactRepository.class, ExpressionUtil.TRUE_EXPRESSION);
IArtifactRepository[] repositories = repoQueryable.query(all, subMonitor.newChild(250)).toArray(IArtifactRepository.class);

The fix for DownloadManager is sorting the array of query result again. Maybe it's better and common to add public interface to get sorted repositories on ProvisioningContext.
Comment 1 Helmut J. Haigermoser CLA 2011-01-24 05:05:29 EST
Ouch, if this is true than 3.6 is severely broken and must be fixed right away,
favoring "http" over "file" repos will kill performance!

Can we get a reply to this ASAP, and bump it to highest priority if your analysis confirms the issue?

Helmut
Comment 2 Pascal Rapicault CLA 2011-01-24 14:21:24 EST
Please provide a test case showing the problem and a patch.
Comment 3 Meng Xin Zhu CLA 2011-01-25 04:27:57 EST
Created attachment 187497 [details]
test case and proposed fix

Let bundle p2.test require the packages related to osgi http service, and add http.port vm argument in the launch.

Let me know what you think. Thanks.
Comment 4 Meng Xin Zhu CLA 2011-01-25 04:30:40 EST
Created attachment 187498 [details]
test data

unzip it, then put the folder under 'testData'
Comment 5 Martin Oberhuber CLA 2011-01-25 06:14:42 EST
CQ:WIND00252176

From what I understand, this is a severe regression in Eclipse 3.6, compared to Eclipse 3.5. The regression affects products that install from both a local repo (like a shipped DVD) and an online repo (like an ESD / update pack site).

Effect of the regression is that the install performance can be severely reduced by downloading artifacts that already exist locally. Customers likely wouldn't accept such a regression, so I'm setting "critical" severity.

I've also updated the summary to show the end-user effect, previous summary was:
DownloadManager.start() doesn't sort the repositories based on the predefined priorities when trying to download the artifacts from the repositories one by one

Please consider the testcase and fix that Meng Xin has provided.
Comment 6 DJ Houghton CLA 2011-01-27 14:25:10 EST
Released a modified patch to HEAD.
Thanks for the investigation.
Closing.
Comment 7 Martin Oberhuber CLA 2011-01-27 15:30:12 EST
Thanks so much, DJ!

Could a backport to 3.6.2 be considered?

We are very close to releasing a product (based on 3.6.x), and this issue is a showstopper for us. For sure we'll have to patch the Platform of our RCP installer ourselves if this doesn't get addressed in mainstream.
Comment 8 DJ Houghton CLA 2011-01-27 15:39:42 EST
Sounds good to me. I've created bug 335610 for this. 
I will attach a patch and then ask a PMC member for a +1.