Community
Participate
Working Groups
Hello *, I've seen that the package plugin repackages every jar during an incremental build. Is there a way to teach tycho that only those jars with changes are repackaged? hint: i trigger the product build with mvn package. is there another command to avoid repackaging? compile kills the creation of the product at the end of the build
I used the past weeks to dig into the matter a little. There is a comment that avoiding repackaging would interfere with the flag that makes repackagin optional. I've created a little method that checks weither the content of the JAR archive changed, and only if so recreates the JAR. In Theory this works out, because when the qualifier had changed, the MANIFEST file is touched and therefore a resource has changed -> JAR is recreated. The Problem is, that the Compiler seems to move Files at random, e.g. Messages.properties files or icons are sometimes recreated in the target, even if they have not changed. Once this issue is fixed, would there be still anything against avoiding repackaging?
finding out whether there was a real change (as opposed to updated timestamps etc) is not easy. You would need detailed knowledge about the types of files and an analyzer for each file type to tell whether a change in the compilation result can be safely ignored. Igor worked on something similar https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers not sure if that's what you want Igor is probably the best to comment here as I know he worked on incremental compilation in another project
I'm curious to know how your use-case compares to https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers . What is it you're trying to avoid/save exactly? Is it only about build time by skipping a repackage step, or more that you'd like the same file to be used if there's no "payload" change? The https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers is intended to cover the 2nd case and works well. It relies on reproducible version qualifier though; but some variations may be possible to implement. (In reply to Moritz Aleithe from comment #1) > I've created a little method that checks weither the content of the JAR > archive changed You may be able to reuse directly ZipComparatorImpl, or another of the ArtifactComparator available in Tycho codebase. > and only if so recreates the JAR. Note that with all the possible way to tweak the packaging (build.properties, pom.xml settings...), it's hard to know before having packaged the jar what it's actually going to be made of. So it's simpler to perform the packaging and then compare jar (produced) with jar (baseline) and do a replacement if the baseline jar is a good output. > The Problem is, that the Compiler seems to move Files at random, e.g. > Messages.properties files or icons are sometimes recreated in the target, > even if they have not changed. That would be a good things to improve this. Can you please open dedicated bugs for the specific issues? I believe fixing them could be profitable to other underlying layers too. > Once this issue is fixed, would there be still anything against avoiding > repackaging? See comment above. Avoiding repackaging is hard because it's hard to really know what the output of the package will really be. A simpler strategy is to do the packaging in some working-copy files, and then only replace the content of target/ if there is a worthy diff to do it. Using a similar strategy as https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers for comparison, but comparing with target/ and before moving files to target/ would be a good thing in general.
(In reply to Mickael Istria from comment #3) > I'm curious to know how your use-case compares to > https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers . > What is it you're trying to avoid/save exactly? Is it only about build time > by skipping a repackage step, or more that you'd like the same file to be > used if there's no "payload" change? The > https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers is intended > to cover the 2nd case and works well. It relies on reproducible version > qualifier though; but some variations may be possible to implement. > My primary concern is the time it takes to recreate the Archive every time. We only use qualifiers on a small subset of our Plug-Ins, but all of our JARs are recreated during incremental build (currently our test system just runs builds, without any changes between two builds, which takes about 20-30 minutes with 12 cores for about 1000 Plug-Ins to build and a TP with about 1000 more bundles) > That would be a good things to improve this. Can you please open dedicated > bugs for the specific issues? I believe fixing them could be profitable to > other underlying layers too. As soon as i can find the "bad guy" who actually does the copying
(In reply to Moritz Aleithe from comment #4) > My primary concern is the time it takes to recreate the Archive every time. > We only use qualifiers on a small subset of our Plug-Ins, but all of our > JARs are recreated during incremental build (currently our test system just > runs builds, without any changes between two builds, which takes about 20-30 > minutes with 12 cores for about 1000 Plug-Ins to build and a TP with about > 1000 more bundles) Ok, and how much time is spent in packaging only out in this 20-30 minutes? > As soon as i can find the "bad guy" who actually does the copying If you think it's the compiler, it could be JDT and you may be able to see same thing from the IDE.
Created attachment 274522 [details] Sampling (without the resolution phase) An nps file generated during the build, except for the resolving part
(In reply to Jan Sievers from comment #2) > finding out whether there was a real change (as opposed to updated > timestamps etc) is not easy. I Think that even rebuilding if the timestamp changed and buying that the change might be irrelevant could help, a further Check would potentially be slower that simply rebuilding the archive
Eclipse Tycho is moving away from this bugs.eclipse.org issue tracker to https://github.com/eclipse/tycho/issues/ instead. If this issue is relevant to you, your action is required. 0. Verify this issue is still happening with latest Tycho 2.4.0-SNAPSHOT if issue has disappeared, please change status of this issue to "CLOSED WORKFORME" with some details about your testing environment and how you did verify the issue; and you're done if issue is still present when latest release: * Create a new issue at https://github.com/eclipse/tycho/issues/ ** Use as title in GitHub the title of this Bugzilla ticket (may include the bug number or not, at your own convenience) ** In the GitHub description, start with a link to this bugzilla ticket ** Optionally add new content to the description if it can helps towards resolution ** Submit GitHub issue * Update bugzilla ticket ** Add to "See also" property (up right column) the link to the newly created GitHub issue ** Add a comment "Migrated to <link-to-newly-created-GitHub-issue>" ** Set status as CLOSED MOVED ** Submit All issues that remain open will be automatically closed next week or so. Then the Bugzilla component for Tycho will be archived and made read-only.
