Community
Participate
Working Groups
Currently if one uses <pomDependencies>consider</pomDependencies> pom dependencies are only considered (and usable) if the have an OSGi-Manifest present otherwise they are silently ignored. This is not only annoying (as one has to download the artifact, check if it is a bundle, then check all dependencies and so on...) but often not feasible to ask the original author to add proper manifest headers and schedule a release just for that purpose. We should therefore add support that missing manifest files are autogenerated and this metadata is the used further down the process. I would propose as a starting point a very simple approach without any configuration options, later on we can enhance this to support even more sophisticated ways to influence the generated manifest.
@Mickael Istria: I have a working prototype for this using BndTools [1] which is dual-licensed Apache License, Version 2.0, or the Eclipse Public License 2.0, did I still need to open a request for this to be added as a new dependency to tycho? [1] https://github.com/bndtools/bnd
I don't think Tycho should automate generation of MANIFEST.MF for 3rd party jars. It would be too intrusive and would create at build-time some artifacts models that don't have anything in common with what's available at run-time and in public repositories. This case of 3rd party libs to OSGi/p2-ify is already handled by multiple technologies: Eclipse EBR (used by Eclipse Orbit), BND Maven plugins, JBoss Tools Locus... I believe it's better that way: keep the OSGi-ification of artifacts separated from plain Tycho build and done as a prerquisite step using one of the technologies mentioned above.
@Mickael Istria: Consequently we then should remove the whole pomDependencies=consider part from tycho completely. As these wrapped bundles behave like already OSGi-fied versions the participate without any problem in the build and even publishing via products or updatesites, I also don't see any "intrusion" here, the changes are completely local to the PomDependencyCollector. Tools like Eclipse EBR and alike simply forces us to take much efforts in additional building/deploying just because tycho dies not support this out-of-the-box even for simple use cases.
(In reply to Christoph Laeubrich from comment #3) > @Mickael Istria: Consequently we then should remove the whole > pomDependencies=consider part from tycho completely. No, because Tycho is used as the underlying technology by some of those "solutions". With Tycho and pomDependencies=consider, it's possible to p2-ify a plain OSGi bundle and include it in some repository easily. Generating p2 metadata seems fine for Tycho because it's the scope (bind p2 and Maven dependency resolution more or less). However, I think generating OSGi MANIFEST.MF is much better handled by bnd. If this issue is fundamental to you, then I suggest you maybe get in touch directly with the OSGi specification development group. If OSGi specifies it can handle plain jars in some clear way, we can then adpat p2 and Tycho to follow those rules. This will be more widely beneficial than introducing this behavior in Tycho, which I see as a workaround.
I would be equally satisfied if tycho would allow to participate other plugins (like bnd) in plugin resolution but afaik this is currently not possible and hard to archive see Bug 393053, Bug 410979, Bug 353889. There are already techniques to use plain jars (bnd for example) its just not possible to use them with tycho without much effort, while supporting these is just a logical consequence for me: Tycho support maven-bundle -> p2 and p2 -> maven resolution, so why not support plain-jar -> maven-bundle as well if its easy to archive. I don't see what OSGi Workingroup can do here, OSGi defines for years an official repository format (OBR) that still is not supported by P2 resolver, so even if there would be a specification (what doesn't make much sense as we are talking about build-time and OSGi is about runtime), how many years should we wait until it would be adapted some day?
Can you please share a scenario where this would be useful? What would happened of the newly generated OSGi artifacts? Would they be published somewhere? That's fundamentally what I'd like to avoid, as it's totally unsafe to publish generated OSGi MANIFEST.MF that were not controlled by some developer. What about non-exported packages? What about the execution environment?...
It would be useful in any scenario where it is useful to use pomDependencies=consider :-) I often encounter one of the following problems: 1) I enable pomDependencies=consider but the dependency itself is not "osgi-fied" -> building the bundle simply failed because the dependency is ignored 2)The dependency itself is a bundle but one of the transitive dependencies is not "osgi-fied", the build might succeed but later in the runtime or product publishing the bundle complains about the missing dependencies. For any non trivial use-case (e.g. just one common-osgi-bundle that mostly is available as a P2 site anyways) I always run into one of these problems when enabling pomDependencies=consider and consequently ending up in creating some kind of "3rd-party-prequisite-build-and-deploy" setup that collects, wraps, deploys everything and the using this P2 Update Site instead of pomDependencies=consider. This renders this feature (for me!) completely useless because in the end I'm often force to do the undesired repacking-as-p2-site anyways (what I try to avoid by using pomDependencies=consider) and can disable the feature afterwards. About EE and packages: This is all derived by BND, weather the result is published somewhere depends on the user (the proposed feature itself does not publish it somewhere anyways), also if he's satisfied with the automatic approach or decides to hand-craft the result anyways. BTW some people even consider BND is doing a better job auto-generating the MANFIEST than most of developer-hand-crafted ones but that's another story :-)
I have searched for a recent example and found the following: Apache Velocity is available as maven bundle [1], also it'd dependencies commons-collections/commons-lang. But the oro dependency [2] is not, and even though it is marked as optional I run into the problem that in my workflow it somewhere was required and then failed with CNFE. As the dependency itself is only used internally by Velocity and I'm only using Velocity API I literally don't care much how "nice" it is packed but I'm still forced to wrap this inside an bundle with own update-site deploy it and so on (alternatively I can deploy it into a maven repro under a new qualifier and/or artifact id of course). [1] https://mvnrepository.com/artifact/org.apache.velocity/velocity/1.7 [2] https://mvnrepository.com/artifact/oro/oro/2.0.8
FWIW, I experimented with a deeper integration between EBR and Tycho some time ago. I think it is doable but there are some hurdles. The closes I found is here: https://github.com/eclipse/ebr/blob/master/releng/maven-plugins/ebr-tycho-extras-plugin/src/main/java/org/eclipse/ebr/tycho/extras/plugin/RecipeBundleProject.java#L36 There are still some missing implementations but it works for EBR to be able to run EBR and Tycho in the same reactor for producing a p2 update site from any Maven POM dependencies. The full EBR builds still take an intermediate step by publishing the generated bundles into local .m2/repository. From there they are consumed by Tycho. I agree with Mickael. I think Tycho should not implement yet another manifest generation approach. Instead it should be made possible to allow a seamless integration with other tools that does it.
(In reply to Gunnar Wagenknecht from comment #9) > Tycho should not implement yet another manifest generation approach. There is currently literally no such approach, so its a bit misleading. Tycho is only consuming existing Manifests in a way completely uncomfortable for the daily usage. I just want Tycho to do what is promises (consider maven dependencies), if it requires to generate an (intermediate) Manifest or whatever I don't mind (in fact Tycho don't need a manifest at all but P2 Metadata, and does a lot more work here as this feature would require to generate this from maven GAV). Instead I'm forced to do intermediate builds/deploys/update-sites/plugins to actually let it "consider" them...
(In reply to Christoph Laeubrich from comment #8) > Apache Velocity is available as maven bundle [1], also it'd dependencies > commons-collections/commons-lang. > But the oro dependency [2] is not So in your final RCP app or p2 repository, you'd actually like to include oro with a generated dummy MANIFEST.MF?
(In reply to Mickael Istria from comment #11) > So in your final RCP app or p2 repository, you'd actually like to include > oro with a generated dummy MANIFEST.MF? In the end yes (even though its a generated and not a dummy Manifest) BND generates currently the following manifest for it: Manifest-Version: 1.0 Ant-Version: Apache Ant 1.5.3 Bnd-LastModified: 1600423454872 Bundle-ManifestVersion: 2 Bundle-Name: oro Bundle-SymbolicName: oro Bundle-Version: 2.0.8 Created-By: 11.0.6 (Azul Systems, Inc.) Export-Package: org.apache.oro.io;uses:="org.apache.oro.text",org.apac he.oro.text;uses:="org.apache.oro.text.regex",org.apache.oro.text.awk ;uses:="org.apache.oro.text.regex",org.apache.oro.text.perl;uses:="or g.apache.oro.text,org.apache.oro.text.regex",org.apache.oro.text.rege x,org.apache.oro.util Originally-Created-By: 1.4.2-b28 (Sun Microsystems Inc.) Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.2))" Tool: Bnd-5.1.2.202007211702 Name: org/apache/oro Implementation-Title: org.apache.oro Implementation-Vendor: Apache Software Foundation Implementation-Version: 2.0.8 2003-12-28 11:00:13 Specification-Title: Jakarta ORO Specification-Vendor: Apache Software Foundation Specification-Version: 2.0.8 If desired, I could even add a header "Tycho-Warning: Auto generated from maven artifact handle with care" ;-)
OK. Wouldn't it be better to contribute directly creation of a good MANIFEST.MF to oro? I'm afraid the desire of making things work kind of conflicts with the goal of making a better world here ;)
(In reply to Christoph Laeubrich from comment #10) > I just want Tycho to do what is promises (consider maven dependencies) Frankly, that statement is a mis-interpretation of a feature. The intention of pomDependencies consider *never* was to make arbitrary Maven dependencies consumable by Tycho. It always came with the requirement that the Maven dependency has proper metadata. I wonder if this an education problem rather than a technical problem. Anyway, I'm not opposed to making it possible for Tycho to *consume* bundles produced by EBR or BndTools within the same reactor. I just don't think it's purely Tycho's task, i.e. some extension in the other plug-ins is ok for a better/deeper integration. Generating manifests on the fly can lead to more problems that it solves. That's why EBR, Orbit and BndTools exist. Is it possible to publish what you have as an extension?
(In reply to Gunnar Wagenknecht from comment #14) > Generating manifests on the fly can lead to more problems that it solves. > That's why EBR, Orbit and BndTools exist. Is it possible to publish what you > have as an extension? Let me give some context here. I think it would be nice to re-use Bnd instructions from repositories such as Orbit and EBR. Thus, if a Maven artifact without metadata is detected external repos can be considered for Bnd instructions. If there are some they could be used. The BSN should probably also all following some custom name scheme to never conflict with official bundles. If no such external metadata is found I would encourage folks to provide one instead of generating a default one, which may lead to unexpected problems at runtime later.
(In reply to Mickael Istria from comment #13) > Wouldn't it be better to contribute directly creation of a good MANIFEST.MF > to oro? I'm afraid the desire of making things work kind of conflicts with > the goal of making a better world here ;) I always do and have provided some patches for OSS adding manifest headers to open source projects but that's not always possible: 1) time : I can't always afford to wait month for a new release 2) no-interest: if the maintainer do not use OSGi the often are not very convinced to add thing they don't know 3) legacy or old version: sometimes for legacy reasons or due to licensing stuff I need to use older packages 4) dependency-hell: adding dependencies is light and easy for maven projects meaning often the above process has to be repeated in several cycles (including waiting for the downstream release, then convince the upstream to upgrade and so on) 5) convenience: I often need to prototype things where I don't know if it will actually makes it into the final product, acting like I can try but first I need to convince project A; B, ... to properly support OSGi in a nice fashion and wait for the next release is often not an option I even try to make the world better for tycho-users wanting to consume plain maven artifacts so adoption of tycho would be even easier ;-) > It always came with the requirement that the Maven dependency has proper metadata. Until now ... its just a limitation of how tycho uses P2/maven for dependency resolution not nature-law. > Generating manifests on the fly can lead to more problems that it solves. sure as every feature, but generating the manifest and the be forced to deploy it somewhere doesn't make it better. I also can't agree with that manually is always better, a good shaped component will generate good default generated Manifest, a poorly shaped one will not become better by manually adding hand-crafted directives. > some extension in the other plug-ins is ok for a better/deeper integration sadly this is currently not possible afaik and support this from a pure bundle is a lot effort. Beside this, we just recently merged tycho+tycho-extras as it has proven no value to manage tightly related things in different repos with separate relases. > I just don't think it's purely Tycho's task No but I think tycho should support at least a simple way to do it as it is meant to integrate OSGi/PDE builds with Maven. For me, this whole "bundle must have metadata" has always felt more like "we don't know better yet" as it was the case with only supporting P2 updatsites a long time. All this Orbit/EBR/whatever stuff surly has it pros but also has it cons if it come to simple or small setups without a whole build-infrastructure where one simply want to use a pre-build artifact and ends up frustrated by simply copy the jar into the project itself :( > I wonder if this an education problem rather than a technical problem. Sorry I don't understand... do you mean we better tell people: Yes we can do it technically but you better learn about OSGi to help it on its tour to world domination? > external repos can be considered for Bnd instructions I don't think another source-of-truth would help here much, as always no one is *forced* to use that feature, and thus I don't see any problem with name-clash or whatever because nothing prevents you to do it equally "wrong" just in an uncomfortable way. Anyways there are demands that bnd file could be used in a tycho-build as well instead of a manifest (see referenced bugs). I must confessed I'm a little bit perplexed that this has lead to such strong repulse of this proposal as it has always have felt this a natural thing to work with maven-artifacts and all this manual repacking/deploying/add just another updatesite/... often feels like a workaround "because tycho can not consume it otherwise". I can understand that for a greater audience its suitable to add more effort (e.g. Orbit Drops) in wrapping things up, but even there I recently have often noticed that things are just evolving faster that the manual (volunteer) work can catch up (if it at least would happen) having rather old versions and compared to increasing a version in a pom incredibly slow upgrade-cycles. Just some examples to stay with that velocity stuff: Orbit has 1.5 which is 13 years(!) old while even the latest release 1.7 is already 10 years old and has not found its way into Orbit!
Here are some constraints I would add if we want Tycho to enable this (ie enable too easily production of crappy OSGi bundles): * The feature should start by being opt-in by default, with a flag like "allowAutomaticOSGiMetadataGenerationForPlainJars" in target-platform-configuration * Tycho itself shouldn't generate the MANIFEST.MF and should defer it to BND or EBR * Generating a MANIFEST.MF should be showing a quite explicit warning * The BSN should be derived from the groupId/artifactId of the consumer and the artifactId of the jar, so if you repack Oro for ChemClipse, the BSN should be something like `org.chemclipse.org.apache.oro`. If all those constraints are met, I am less anxious about the idea that Tycho starts producing bad artifacts too magically.
Thanks for your patience, I think I can offer a solution that fulfills most of these requirements. As BND is required either way (EBR uses BND under the hood) should I simply supply a review for this or do we need to request clearance from the IP team for using BND? Im just a little bit uncertain about "the BSN should be derived from the groupId/artifactId of the consumer" as - What is meant by consumer? the "pom" that declares the dependency? The bundle that is currently resolved? - If multiple poms/bundles refers to the same dependency how to resolve this? Would you mind relax the requirement to name it with a fixed sceme like <groupId>.<artifactId>.wrapped? I could even think about making this configurable so one could choose to have prefix/suffix as desired.
(In reply to Christoph Laeubrich from comment #18) > - What is meant by consumer? the "pom" that declares the dependency? The > bundle that is currently resolved? Yes. > - If multiple poms/bundles refers to the same dependency how to resolve this? I don't think there is a special case to handle. The OSGi bundle with MANIFEST should be generated for each dependency on each module, with and opportunity of caching them to save time. In the end, the one that matters in the delivery is the one that will be generated and included in the resulting p2 repo and/or RCP application. Other ones would just be transient disposable files. > Would you mind relax the requirement to name it with a fixed sceme like > <groupId>.<artifactId>.wrapped? At the moment, I don't really want to relax things. It needs to be clear by the name of the file which project has generated it and is responsible for it. I think <consumerGroupId>.<dependencyGroupId>.<dependencyArtifactId> is a must-have so far. Keep in mind that in my opinion, this feature can create a lot of bad OSGi bundles or try to OSGi-fy without tests code that's not really good with OSGi (context classloader...), I don't want to give the impression that this is Tycho's fault if such bad artifacts spread everywhere, it has to be clearly the responsibility of the producer who agreed to publish those artifacts by setting explicitly a property and ignoring the warnings at build time. > I could even think about making this configurable so one could choose to > have prefix/suffix as desired. I think it's better to enforce a naming pattern that has some strong sense than to let consumers just force confusing name that would seem generic and "official" distribution of libraries.
But think about bundle a (bsn test1) and bundle 2 (bsn test2) will both require maven artifact parentId.artifactId then there will be two bundles generated with the same content but different name test1.parentId.artifactId test2.parentId.artifactId that doesn't make sense or I completely misunderstand how it should work it would even be impossible to include these in a feature as it will generate a new name for the same artifact. btw Apache Karaf is a widely used project that support this wrapping pattern out-of-the-box and beside the happy users I have never heard about anyone blame karaf for generating "bad bundles". So this fear atm just feels a bit odd and constructed to me as I can easily generate a bundle named whatever I like and upload it somewhere and we do not enforce any naming sceme on them for traditional bundles e.g. I can create and build bundles that are named org.eclipse.tycho.* without a problem... So the only way I see this can work is either we simply use the artifactId as it is done in the whole tycho code, a fixed scheme or let it be configurable for those who are certain about confusion because they like to publish this for a greater audience.
(In reply to Christoph Laeubrich from comment #20) > But think about bundle a (bsn test1) and bundle 2 (bsn test2) will both > require maven artifact parentId.artifactId then there will be two bundles > generated with the same content but different name > > test1.parentId.artifactId > test2.parentId.artifactId In my previous comment, I mentioend <consumerGroupId>.<dependencyGroupId>.<dependencyArtifactId> ; so this would be something like `org.chemclipse.org.apache.org`, and this would be the same for all modules that have the same groupId. > btw Apache Karaf is a widely used project that support this wrapping pattern > out-of-the-box and beside the happy users I have never heard about anyone > blame karaf for generating "bad bundles". How are those bundles identified as being built by Karaf? What happens if some other project does create and publish similar wrappers with same name (and maybe even subtle differences). > So this fear atm just feels a bit odd and constructed to me as I can easily > generate a bundle named whatever I like and upload it somewhere and we do > not enforce any naming sceme on them for traditional bundles e.g. I can > create and build bundles that are named org.eclipse.tycho.* without a > problem... Right, and it's one issue; if we can avoid spreading such bad practices, that's good. > So the only way I see this can work is either we simply use the artifactId > as it is done in the whole tycho code, a fixed scheme or let it be > configurable for those who are certain about confusion because they like to > publish this for a greater audience. Please use the groupId of the consumer as prefix.
(In reply to Mickael Istria from comment #21) > How are those bundles identified as being built by Karaf? What happens if > some other project does create and publish similar wrappers with same name > (and maybe even subtle differences). They simply aren't marked anything special, wrapping a bundle only adds meta-data and in most cases there is no need to deploy them anywhere. Its more that maven is the source of truth and one simply has no demand to consume it from somewhere else. I just wanted to point out that there are case where automatic wrapping is used regularly without bad things happen every day. On the other side, the lot of effort to wrap all and everything (being it an bundle or not), inside an P2 update side could be regarded a "bad thing", and theres always a world outside eclipse ;-) > Please use the groupId of the consumer as prefix. I'll try this then so we can make progress here...
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/169618
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/169618 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=adf2b8a34be5b8337de7619622569238fdc58bdf
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/169844
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/169844 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=bfa517242726f9f8825fb12fdafef8c5924be142
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/169894 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=64217e6e2c7a3c36bec40f9ab15d8b0896a051eb
So this is merged, thanks Christoph! I'm leaving it open because I think we still need: * Some documentation in the release notes * Probably some better documentation on the mojo and enum (that would surface inside IDE in maven docs) * 1 integration test that could serve as example of how things are supposed to work and be used
I have added N&N [1] now, are those docs generated from the javadoc of the class or where need those be placed to be picked up by IDE/maven doc? [1] https://wiki.eclipse.org/Tycho/Release_Notes/2.1#Automatic_generation_of_OSGi_metadata_for_maven_dependencies
(In reply to Christoph Laeubrich from comment #29) > I have added N&N [1] now, are those docs generated from the javadoc of the > class or where need those be placed to be picked up by IDE/maven doc? They're supposed to be extracted from Javadoc, so I think we're good from a documentation POV.
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/170376
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/170376 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=133e36abba53869ac41a9868a931781f355d87f3
Just as an info for everyone trying to use Maven artifacts in both Tycho and PDE: You can also do that using https://github.com/reficio/p2-maven-plugin to automatically convert plain jars to p2 update sites.
(In reply to Michael Keppler from comment #33) > Just as an info for everyone trying to use Maven artifacts in both Tycho and > PDE: You can also do that using https://github.com/reficio/p2-maven-plugin > to automatically convert plain jars to p2 update sites. I hope and I believe this addition in Tycho will make such plugin useless. I wouldn't recommend anyone to start using it now Tycho has a good built-in feature to p2-ify random Maven artifacts.
