Community
Participate
Working Groups
Created from: TYCHO-530: Support p2.statsURI https://issues.sonatype.org/browse/TYCHO-530 Original Description: As an eclipse plugin developer, I am interested in finding out how many times my plugin is downloaded. When generating an update site, I would appreciate being able to define: * a p2.statsURI entry; * a collection of plugins for which stats should be tracked. While these can be tracked e.g. by the HTTP server access logs, many open source hosting services, notably SourceForge.net , do not make these logs available. Rather, it is up to the hosted project to track web page hits using analytics services like Google Analytics. But obviously these do not work for p2 clients. Therefore this is the only solution for such projects. Eclipse references: http://wiki.eclipse.org/Equinox_p2_download_stats . There are also some tycho references at https://bugs.eclipse.org/bugs/show_bug.cgi?id=310132 . Original Comments: == jan.sievers == [1290512357000] Hugues Malphettes did some download stats work in his tycho fork https://github.com/hmalphettes/sonatype-tycho/commit/4a9ab45aca13dc8fed22915903089d684f09565d which we may consider.
We'll first need to get the Tycho sources to Eclipse (for the IP/contribution process), but then we could apply Hugues' ideas. Note however that Hugues will need to contribute his changes under the eclipse.org terms and conditions [1], e.g. by attaching the change as patch to this bug. We can't just copy code from anywhere. [1] http://www.eclipse.org/legal/termsofuse.php
As far as I understand, this is a follow-up to bug 310132. Once that bug is resolved, Tycho needs to be able to pass the new parameters, correct?
Will the referenced bug support the @eclipse-update_site@ packaging as well?
(In reply to comment #3) > Will the referenced bug support the @eclipse-update_site@ packaging as well? I won't be investing in eclipse-update-site. IMHO only eclipse-repository has a future in Tycho (see bug 342876).
FYI, we've implemented addition of p2 stats metadata to an eclipse-repository as part of a big "GenerateRepositoryFacadeMojo". You can see it here: https://github.com/mickaelistria/jbosstools-maven-plugins/blob/JBIDE-11065/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/GenerateRepositoryFacadeMojo.java#L356 Any feedback/contrib is welcome. Also this is still a SNAPSHOT, but it will for sure get published on JBoss Nexus if you want to consume it directly.
(In reply to comment #5) > FYI, we've implemented addition of p2 stats metadata to an > eclipse-repository as part of a big "GenerateRepositoryFacadeMojo". You can > see it here: > https://github.com/mickaelistria/jbosstools-maven-plugins/blob/JBIDE-11065/ > tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/ > sitegenerator/GenerateRepositoryFacadeMojo.java#L356 > > Any feedback/contrib is welcome. > Also this is still a SNAPSHOT, but it will for sure get published on JBoss > Nexus if you want to consume it directly. see also p2 bug 401960 for a plan of long-term solution in p2. As plans go, I guess we will live with other more pragmatic approaches for the time being such as Mickaels mojo.
(In reply to comment #5) > FYI, we've implemented addition of p2 stats metadata to an eclipse-repository as > part of a big "GenerateRepositoryFacadeMojo". You can see it here: > https://github.com/mickaelistria/jbosstools-maven-plugins/blob/JBIDE-11065/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/GenerateRepositoryFacadeMojo.java#L356 > > Any feedback/contrib is welcome. > Also this is still a SNAPSHOT, but it will for sure get published on JBoss Nexus > if you want to consume it directly. What's the best place to have a discussion? I'm interested in consuming this in the mylyn-mantis plugin.
Good question... Send me private mails, I'll give you help and use your questions to populate the wiki.
FWIW, I was able to add "download.stats" properties to bundles and/or features in *content.xml* using p2.inf files (META-INF/p2.inf for bundles, p2.inf for features) [1]. I wasn't able, however, to add a "p2.statsURI" properties to the (eclipse-)repository at all. Also, none of the bundle or feature properties show up in the artifacts.xml where they are suppossed to be according to [2]. So, my question is this: Are p2.inf properties supposed to end up in content.xml only? If not and this could be fixed in Tycho, we would be able to support p2.statsURI quite easily through the p2.inf mechanism. [1] <http://wiki.eclipse.org/Equinox/p2/Customizing_Metadata#Property_Advice:> [2] <http://wiki.eclipse.org/Equinox_p2_download_stats#Enabling_stats_in_your_repository>
(In reply to comment #9) > So, my question is this: Are p2.inf properties supposed to end up in content.xml > only? If not and this could be fixed in Tycho, we would be able to support > p2.statsURI quite easily through the p2.inf mechanism. I think the problem here again is that the publishers only publish into the modules' p2 repositories, and the eclipse-repository build result (target/repository/*) is created by coping form the modules' repositories. So again (as with discovery URLs), the right approach would be to directly configure them in the pom.xml (tycho-p2-repository-plugin) and have that mojo insert them into the result. BTW, this could be a fairly simply patch, why don't you give it a try [1]? [1] http://wiki.eclipse.org/Tycho/Contributor_Guide
Is there a way to specify a "format" attribute tot he tycho-p2-repository-plugin to pass a template site containing some additional properties, just like it is allowed for the p2 ant tasks? http://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fp2_repositorytasks.htm
No, there isn't [1]. But rather than copying properties from another repository, we should have the option to specify properties in the POM. This seems more like the way you'd implement this in Maven... [1] http://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-repository-plugin/assemble-repository-mojo.html
(In reply to comment #12) > No, there isn't [1]. But rather than copying properties from another > repository, we should have the option to specify properties in the POM. This > seems more like the way you'd implement this in Maven... +1. I also think that the approach of using a "format" as template repository in p2 Ant tasks is too complicated compared to directly setting key/value pairs.
I'm thinking that it would be better if it could be a <properties> tag to configure the tycho-p2-plugin, so it would customization of p2 properties on any packaging-type and not only eclipse-repository. WDYT?
Having native Tycho support for arbitrary(?) properties is certainly preferable, but there at least exists a workaround using the tycho-eclipserun-plugin [1]. That workaround has IMHO only one small drawback (besides working only for p2.statsURI and p2.mirrorsURL): The tycho-eclipserun-plugin:eclipse-run goal runs *after* tycho-p2-repository-plugin:archive-repository so the archived repository does *not* contain the added p2.statsURI and p2.mirrorsURL properties. Unfortunately, fixing this is hard, as Tycho binds pretty much every goal to the same phase: "package". If Tycho would use "prepare-package" [2] as intended, then setting up eclipse-run to run at the end of "prepare-package" and before archive-repository in "package" would be feasible. [1] <http://www.codetrails.com/blog/maven-tycho-how-to-configure-your-repos-mirror-and-statistics-uris> [2] <http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference>
(In reply to comment #15) > If Tycho would use "prepare-package" [2] as > intended, then setting up eclipse-run to run at the end of "prepare-package" and > before archive-repository in "package" would be feasible. I have been thinking about this as well, but we definitely need a separate enhancement request for this.
(In reply to comment #14) > I'm thinking that it would be better if it could be a <properties> tag to > configure the tycho-p2-plugin, so it would customization of p2 properties on any > packaging-type and not only eclipse-repository. > WDYT? I think you lost me: What kind of properties are you talking about? Repository properties or properties on the units/artifact entries? Setting repository properties on the modules other than eclipse-repository doesn't make sense because the properties of module repositories are never read (and in fact not even persisted).
Any updates here? Download stats get more prominent in Mars M1 with direct PDE support in category.xml [1]. [1] http://download.eclipse.org/eclipse/downloads/drops4/S-4.5M1-201408062000/news/#category-download-statistics
Just to explain why download stats add value on top of plain web server access logs: One can include additional data about e.g. distribution package, version, platform etc. as parameters [1]. [1] http://wiki.eclipse.org/Equinox_p2_download_stats
(In reply to Christian Georgi from comment #18) > Any updates here? > Download stats get more prominent in Mars M1 with direct PDE support in > category.xml [1]. > > [1] > http://download.eclipse.org/eclipse/downloads/drops4/S-4.5M1-201408062000/ > news/#category-download-statistics If support for download stats was added to category.xml, the p2 category publisher taht Tycho uses should add the required p2 metadata. This would mean as soon as we switch tycho to use p2 version Mars M1 or better, you should get this feature in Tycho.
(In reply to Jan Sievers from comment #20) > This would mean as soon as we switch tycho to use p2 version Mars M1 or > better, you should get this feature in Tycho. Can you do so with Tycho 0.22?
for reference, p2 bug 436872 ( commit https://github.com/eclipse/rt.equinox.p2/commit/9ea8e4cc9aa6ec8ff70a4e0a811401f5c941ad68 ) implements p2.statsURI in category.xml with Mars M1 > Can you do so with Tycho 0.22? yes, we should consume a Mars version of p2 with Tycho 0.22 . Probably a later Mars milestone though.
https://git.eclipse.org/r/#/c/33226/
(In reply to comment #20) > This would mean as soon as we switch tycho to use p2 version Mars M1 or better, > you should get this feature in Tycho. AFAIK, this won't work in Tycho out of the box. The reason for this is that we currently publish the category.xml into the module repository (i.e. p2-metadata.xml) and not into the /target/repository repository. Module repositories don't support repository metadata, so any such information generated by publishers is dropped. It doesn't make sense to add support for repository metadata in module repositories because then it would be unclear how that metadata should be copied into the final result. However we could change the category.xml publishing to directly write into /target/repository. AFAIK this wouldn't break anything - categories don't need to be available to other modules in the build - and would allow us to benefit from the changes in the category.xml publisher.
(In reply to Tobias Oberlies from comment #24) > publishing to directly write into /target/repository. AFAIK this wouldn't > break anything - categories don't need to be available to other modules in > the build - and would allow us to benefit from the changes in the > category.xml publisher. Make it so :)
(In reply to Tobias Oberlies from comment #24) > (In reply to comment #20) > AFAIK, this won't work in Tycho out of the box. The reason for this is that > we currently publish the category.xml into the module repository (i.e. > p2-metadata.xml) and not into the /target/repository repository. Module > repositories don't support repository metadata, so any such information > generated by publishers is dropped. OK, that explains why it didn't work with https://git.eclipse.org/r/#/c/33226/
also note that there is a newly introduced "mirrorProperties" switch (defaults to false) in the p2 mirror application https://github.com/eclipse/rt.equinox.p2/commit/9ea8e4cc9aa6ec8ff70a4e0a811401f5c941ad68 so we maybe still use stats properties when mirroring.
(In reply to Jan Sievers from comment #27) > so we maybe still use stats properties when mirroring. s/use/lose
mirrorProperties are currently ignored in the same way as the stats. These would be supported as well if we published directly into /target/repository.
implementation turns out to be tricky (maybe Tobias you can add some comments why/what are the problems), so looks to me like this won't make it into 0.22.0
(In reply to comment #29) > if we published directly into /target/repository. Do you know if someone already tried to publish the category.xml directly into SimpleMetadataRepository/SimpleArtifactRepository instances writing to /target/repository? I don't seem to remember any particular problems with that approach...
(In reply to Tobias Oberlies from comment #31) > (In reply to comment #29) > > if we published directly into /target/repository. > > Do you know if someone already tried to publish the category.xml directly > into SimpleMetadataRepository/SimpleArtifactRepository instances writing to > /target/repository? I don't seem to remember any particular problems with > that approach... I had tried to hack it and failed. I lost the local commit in my last hardware exchange so need to try and upload a new one so we can better discuss the problem in gerrit. IIRC stats properties on repository level were added but stats properties on each individual artifact were missing. I remember vaguely we discussed this at EclipseCon and you mentioned this would not work because artifact metadata must be local to the module being published and not appended at the very end of the build when aggregating the repository.
(In reply to comment #32) > IIRC stats properties on repository level were added but stats properties on > each individual artifact were missing. I remember vaguely we discussed this at > EclipseCon and you mentioned this would not work because artifact metadata must > be local to the module being published and not appended at the very end of the > build when aggregating the repository. You are right - thanks for refreshing my memory ;-) So we have two options for the "download.stats" properties needed in the artifact descriptors (=artifact.xml entries): * We could make the "enable stats" a POM configuration that can be switched on or off on each artifact individually. However this implies that the "enable stats" could never be changed for external artifacts - they would always have stats on or off in the same way as they were on the remote p2 repository. * Bite the bullet and hack the artifact descriptors according to the category.xml configuration in an p2 repository afterburner step. The way that p2 stats concept is, I have doubts that they can work in a modular build system without hacks.
Hi, as Tycho now (as of Bug 471693) produces XZ-compressed artifacts.xml.xz files in addition to the regular artifacts.jar, one workaround to *this* bug became infeasible: Using the tycho-eclipserun-plugin to run org.eclipse.wtp.releng.tools.addRepoProperties to post-process the artifacts.jar. As Tycho automatically creates the artifacts.xml.xz file as part of its tycho-p2-repository-plugin:archive-repository goal, any post-processor would now need to process two files instead of one -- which at least the addRepoProperties does not. We (Code Recommenders) just got bitten by this, producing inconsistent artifacts metadata (Bug 497546). :-( I wonder whether Tycho's archive-repository should be split in two, one producing the raw artifacts.jar (or even .xml) and one producing the XZ-compressed version? That way, users could squeeze in goals like tycho-eclipserun-plugin:run. Alas, there's the obvious lack of lifecycle phases after "package" for this purpose, so squeezing in another goal between the defaults requires some ugly POM element ordering hackery.
bug 539552 covers addition of download.stats in artifact metadata as a setting on the tycho-p2-plugin:generate-metadata goal. For p2.statsURI, we don't need to focus on stats-specific case. It's "only" about adding a property to the artifact repository. I suggest that we just introduce an <extraArtifactRepositoryProperties> parameter to the tycho-p2-plugin:assemble-repository, and anyone could add the <p2.statsURI>blah</p2.statsURI> as a property. If we do it that way on the right layer, it would cascade to this propagated in artifacts.jar and artifacts.xml.xz and whatever is derived from the actual repository.
(In reply to Mickael Istria from comment #35) > we don't need to focus on stats-specific case Please do p2.mirrorsURL too.
(In reply to Mickael Istria from comment #35) > bug 539552 covers addition of download.stats in artifact metadata as a > setting on the tycho-p2-plugin:generate-metadata goal. > For p2.statsURI, we don't need to focus on stats-specific case. It's "only" > about adding a property to the artifact repository. > > I suggest that we just introduce an <extraArtifactRepositoryProperties> > parameter to the tycho-p2-plugin:assemble-repository, and anyone could add > the <p2.statsURI>blah</p2.statsURI> as a property. > If we do it that way on the right layer, it would cascade to this propagated > in artifacts.jar and artifacts.xml.xz and whatever is derived from the > actual repository. Sounds pretty good and open for any properties.
The extra properties should be parameters for AssembleRepositoryMojo, Most likely forwarded in the DestinationRepositoryDescriptor, Consumed in the MirrorApplicationServiceImpl.createMirrorApplication() as a parameter in constructor of org.eclipse.tycho.p2.tools.mirroring.MirrorApplication so it can be used in its initializeDestination() method.
New Gerrit change created: https://git.eclipse.org/r/130318
New Gerrit change created: https://git.eclipse.org/r/130326
Gerrit change https://git.eclipse.org/r/130318 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=7adf81e731490de4e852ae27cca05b4228a05c23
Gerrit change https://git.eclipse.org/r/130326 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.extras.git/commit/?id=96018f22b1bd9317f5953596dd549ca1268f402a
Done. See https://wiki.eclipse.org/Tycho/Release_Notes/1.3#Extra_artifact_repository_properties_.28like_p2.statsURI_or_p2.mirrorsURL.29
