Community
Participate
Working Groups
Currently Tycho does resolve and *download* dependencies very early during the build. While the former (resolution from p2 metadata) isn't easy to avoid, and actually doesn't take too much time; actually *downloading* the artifacts can take very long while not being useful in some cases (eg `mvn validate` or even just `mvn compile` which don't need all transitive deps). Tycho should investigate lazy download of the artifacts. Note: this issue is different from bug 353889 as this current is only about lazily downlaoding artifacts, while bug 353889 is about changing the orchestration of dependency resolution.
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171206
So it appears this one is extremely difficult to me. It's relatively easy to tweak the p2 based dependency resolution to avoid an early download of artifacts, so we can delay the download a bit later. However, a 2nd pass of resolution takes place just after, and this one uses the EquinoxResolver and OsgiBundleProject.getClasspath() elements; which do need the artifacts locally as this pass reads MANIFEST.MF. I've tried to delay a bit the computation of classpath, but it still happens early. Overall, what I'm not getting at the moment is why is there even a 2nd pass of resolution? What's missing in the 1st p2-based pass which requires to them read MANIFEST.MF for all target platforms element so early... I've put my experiments at https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171206 . What need to be investigated is whether the whole build path and 2nd resolution pass has to happen very early. However I think this work begins to look like a duplicate of bug 353889 and Tobias' comment on that other bug are totally in sync with what would be expected to happen here (1st: compute a build order, then start building all modules with extra resolution if necessary, but not so early).
P2 offers only *runtime* dependency resolution while the later provides *compile time* dependency. For example it is possible to place jars or even pre-compiled classes inside a bundle. I'm a bit confused about that verify/compile thing because I would at least at that stage assume that the full classpath has to be available? So I wonder at what phase one would expect the download? This might also be a limitation of maven itself, as from my experience maven itself starts to "download the internet" whenever I try to compile a new project very early. What would be crucial from my point of view to investigate this further would be to have one simple case (preferable just one bundle) that shows one specific case/build command where an artifact is downloaded "to early" and we should give an idea why it should not be necessary to download it. Then it would be possible to debug why/when it is downloaded and if this might really be optional, for example if it is just for meta-data that is already available via p2 we might need to insert a link here in between to transfer/convert the data into necessary formats.
(In reply to Mickael Istria from comment #2) > So it appears this one is extremely difficult to me. Just wondering, even if I can understand the purpose for this, is this really such a big problem as I can hardly think about use cases where the artifacts are not needed (beside the usual 'mvn clean' usecase).
(In reply to Christoph Laeubrich from comment #4) > (In reply to Mickael Istria from comment #2) > > So it appears this one is extremely difficult to me. > > Just wondering, even if I can understand the purpose for this, is this > really such a big problem as I can hardly think about use cases where the > artifacts are not needed (beside the usual 'mvn clean' usecase). See the bug marked as dependent: we cannot at the moment verify that a build plan is satisfiable without downloading all artifacts; so the feedback about satisfiable plan does happen too late. For the case of an eclipse-plugin I'd expect the dependencies to be downloaded during compile. For the case of an eclipse-repository, I'd expect the dependencies to be downloaded when aggregating. In case of SimRel, it would be `mvn validate` verifying the dependency resolution, without downloading any artifact. Basically, Tycho should keep a reference to the InstallableUnit as long as possible and only fetch the underlying artifact when an operation actually needs the file. If we can just enable this for eclipse-repository, it could actually be enough for the main use-case (SimRel). Maybe the submitted patch may actually already do it. I'll need to try later.
(In reply to Mickael Istria from comment #5) > See the bug marked as dependent: we cannot at the moment verify that a build > plan is satisfiable without downloading all artifacts; so the feedback about > satisfiable plan does happen too late. > For the case of an eclipse-plugin I'd expect the dependencies to be > downloaded during compile. 'mvn validate' should not download and 'maven compile' should do it right? Is there an easy reproducer for this or anything special or can I simply use any project that contains at least one eclipse-plugin module? > Basically, Tycho should keep a reference to the InstallableUnit as long as > possible and only fetch the underlying artifact when an operation actually > needs the file. I think all information is already there we just need to use it, I'll try to also take a look at this I just a bit short of time atm.
(In reply to Christoph Laeubrich from comment #6) > 'mvn validate' should not download and 'maven compile' should do it right? Yes, although at the moment I think we can restrictthe use-case to eclipse-repository: dependency resolution remains happening early "afterProjectRead" but download of necessary deps would only happen during execution of the assemble-repository mojo. Not that this late download may also allowto skip downloads.of useless transitive dependencies in some cases. > > Is there an easy reproducer for this or anything special or can I simply use > any project that contains at least one eclipse-plugin module? No simple reproducer yet. I'll work on it this Thursday if everything remainsas expected. > I think all information is already there we just need to use it, Indeed, my current experiments align with that: data is already there and this change doesn't require more things, it's only a matter of just replacing some direct access to dependency files in early lifecycle by aome strategy to get the file only when necessary. > I'll try to > also take a look at this I just a bit short of time atm. Thanks, but no need for you to worry or rush too much about it, as I think I knowhow to achieve the goal here and I have a sufficient "time budget" for it.
Bug 372780 seems to be a prerequisite: currently the dependencies needs to be fetched and resolved early to be added as dependencies in the Maven model (like <dependency>). Allowing to delay downloads means that we need to delay or avoid creation of those dependencies unless necessary. If we stop relying on them in Tycho specific goals, that would make things much easier.
I wonder if this is even supported by maven. If I start with an empty local repository and perform "mvn clean" will the artifact of a dependency be downloaded? What about the meta-data (pom/md5)? That resembles mostly what P2 provides.
(In reply to Christoph Laeubrich from comment #9) > If I start with an empty local > repository and perform "mvn clean" will the artifact of a dependency be > downloaded? What about the meta-data (pom/md5)? That resembles mostly what > P2 provides. Metadata would be downloaded (as they're necessary for dependency resolution to compute the build graph) and cached for further reuse. Artifacts wouldn't be downloaded until necessary (ie until assemble-repository is called)
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/171206 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=f9eadda96c46cc35bddf553ed7c810c8b73d1284
Added N&N entry: https://wiki.eclipse.org/Tycho/Release_Notes/2.2#Delay_download_of_p2_artifacts_for_non-plugin_modules
*** Bug 375111 has been marked as a duplicate of this bug. ***
