Community
Participate
Working Groups
The latest Orbit is this: http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/ But there's no link to the p2 update site, which is here: http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/updateSite/ In order to encourage people to start using the p2IU format and/or to start installing Orbit bundles with p2.director directly, why not tell people this update site exists? I've tested the site and it works quite will with p2.director for platform provisioning: http://wiki.eclipse.org/Athena_Progress_Report#2009-09-01
+1 I'd also like to throw in my weight again for a simpler URL to the p2 repository. http://download.eclipse.org/tools/orbit/recommended/ http://download.eclipse.org/tools/orbit/stable/ http://download.eclipse.org/tools/orbit/integration/ The latter one could just be a link which performs a HTTP redirect to the latest integration build update site. The "/recommended"" could actually be a composite repository.
+1 In Indigo M4 CDO seemed to cause trouble by not having used the latest URL ;-(
There was recent comments about this concept in bug 310578#c14 and some on cross-project list. There appears to me to be a mix of use cases that involve release engineers doing builds, and some that involve developers using target runtimes. Perhaps the end-result or solution is exactly the same ... but, gives me pause. But in either case, this seems to be such a highly desired feature, it seems a good case to ask for volunteers. I've been reluctant to do anything since I am maxed out on releng tasks and I could imagine things like this having issues down the road that are as of yet unknown (such as cleanup, archives, conflicts, etc). So, my proposal is for someone to step forward to _own_ the production and on-going use and development of these common repos and to do it in such a way that the current builds and build-repos all stay exactly as they are. I could envision this being solved by some completely separate process, that used the build repos as input, and mirror the results to where there would want, either directly, or as composites. And could be done in a way that was largely, if not completely, independent of current Orbit builds. So, the volunteer must commit to not only initial setup, writing scripts, getting reviews and buy-in, but also maintain and address issues on a long term basis. So, no fighting over it ... if there are multiple volunteers, I'm sure we can work that out. Thanks,
(In reply to comment #3) > There was recent comments about this concept in bug 310578#c14 and some on > cross-project list. There appears to me to be a mix of use cases that involve > release engineers doing builds, and some that involve developers using target > runtimes. Perhaps the end-result or solution is exactly the same ... but, gives > me pause. > > But in either case, this seems to be such a highly desired feature, it seems a > good case to ask for volunteers. I've been reluctant to do anything since I am > maxed out on releng tasks and I could imagine things like this having issues > down the road that are as of yet unknown (such as cleanup, archives, conflicts, > etc). > > So, my proposal is for someone to step forward to _own_ the production and > on-going use and development of these common repos and to do it in such a way > that the current builds and build-repos all stay exactly as they are. I could > envision this being solved by some completely separate process, that used the > build repos as input, and mirror the results to where there would want, either > directly, or as composites. And could be done in a way that was largely, if not > completely, independent of current Orbit builds. > > So, the volunteer must commit to not only initial setup, writing scripts, > getting reviews and buy-in, but also maintain and address issues on a long term > basis. > > So, no fighting over it ... if there are multiple volunteers, I'm sure we can > work that out. > > Thanks, A composite p2 repository is a good solution. I use it right now in several projects. Hosted and maintained under /downloads/modeling/tmf/xtext/updates/orbit/
*** Bug 409255 has been marked as a duplicate of this bug. ***
I've created a cron job that updates the following p2 composite repos in 5 minute intervals: http://download.eclipse.org/modeling/emf/cdo/orbit/latest-R http://download.eclipse.org/modeling/emf/cdo/orbit/latest-S http://download.eclipse.org/modeling/emf/cdo/orbit/latest-I (currently empty) http://download.eclipse.org/modeling/emf/cdo/orbit/latest-M (currently empty) Each composite contains at most one child repo which is the latest Orbit repo of the respective build type, i.e., R, S, I or M-build. The p2.timestamp of a composite only changes when the child location changes. Please feel free to use these stable URLs yourself if you find them handy. If there's interest to run the job under Orbit's identity and create the composites somewhere under .../tools/orbit I'd be happy to contribute the involved scripts for a review.
(In reply to Eike Stepper from comment #6) > I've created a cron job that updates the following p2 composite repos in 5 > minute intervals: > > http://download.eclipse.org/modeling/emf/cdo/orbit/latest-R > http://download.eclipse.org/modeling/emf/cdo/orbit/latest-S > http://download.eclipse.org/modeling/emf/cdo/orbit/latest-I (currently > empty) > http://download.eclipse.org/modeling/emf/cdo/orbit/latest-M (currently > empty) > > Each composite contains at most one child repo which is the latest Orbit > repo of the respective build type, i.e., R, S, I or M-build. The > p2.timestamp of a composite only changes when the child location changes. > > Please feel free to use these stable URLs yourself if you find them handy. > > If there's interest to run the job under Orbit's identity and create the > composites somewhere under .../tools/orbit I'd be happy to contribute the > involved scripts for a review. Sure, there's interest. "helpwanted" being marked for a few years now. Finally! :) I can give a few comments based on your words, but if you'd like to pursue, and maintain, I'm sure we'd entertain making you a "releng committer". a) 5 minutes seems way too often. Most of these repos change at most once a week, sometimes, once only every few months. b) Ideally, the "make composite" script would be invoked at the end of the "promote.sh" script ... since that's the one that moves something from "build machine" to "downloads". Sounds great though. Thanks for your initiative.
> I can give a few comments based on your words, but if you'd like to pursue, > and maintain, I'm sure we'd entertain making you a "releng committer". Thanks but I'm not sure that this would be necessary ;-) > a) 5 minutes seems way too often. Most of these repos change at most once a > week, sometimes, once only every few months. True, but when that happens we might want to see the result quickly. The scripts finish veeeery quickly in either case, with or without changes detected. > b) Ideally, the "make composite" script would be invoked at the end of the > "promote.sh" script ... since that's the one that moves something from > "build machine" to "downloads". Yeah, that's what I thought, too. But my approach was the only one I could implement/test without being a committer. You've probably noticed that I've attached the needed (very short) bash scripts to my posting on cross-project-issues. What do you think how it should go on?
(In reply to Eike Stepper from comment #8) > You've probably noticed that I've attached the needed (very short) bash > scripts to my posting on cross-project-issues. What do you think how it > should go on? I tried looking at them, but couldn't really intuit what they were doing, so I'll just ask. Are they actually "combining" several repositories, into one? (via composite) Or, is it just a way to have "one name" or one URL for what ever is the latest one? The former would be problematic for a number of reasons. The later I could sort of see as helpful to some people ... though I'd worry about "things changing" and the consumer (builder) not really knowing about it.
Oh, and meant to say, last I heard, contrary to anyone's intuition, p2 does not actually make use of the p2 timestamp ... maybe you know of a builder that does? But I think p2 itself simply goes by "date on file system" (or, what ever the http counter part is called).
(In reply to David Williams from comment #9) > Are they actually "combining" several repositories, into one? (via composite) No. > Or, is it just a way to have "one name" or one URL for what ever is the > latest one? Yes. It's like a p2 symlink to the one latest repo ;-) > The former would be problematic for a number of reasons. > > The later I could sort of see as helpful to some people ... though I'd worry > about "things changing" and the consumer (builder) not really knowing about > it. I see it most useful for the S, I, and M-builds, not so much the R-builds.
Thanks for the explanations. And, I really do feel like Orbit is the place for these ... otherwise, you are basically "providing Orbit bundles, outside of Orbit" and while I'm sure it'd be correct/same nearly all the time ... there is always some risk someone would get the wrong thing and mess up the release train -- and then it's basically your responsibility! :) [I admit this would be rare in practice, but ... I'm just saying there is a reason things are divided up into projects ... i.e. it is Orbit that decides that to provide to the release train, and by you providing a different link, then it is you making that decision, conceptually]. If you were willing to do "in Orbit", I'd like to see it done/changed as follows: 1) Use p2 APIs (such as ant task, or p2 application) to create/remove things from the composite, instead of some "manually" written XML ... not a big deal ... just seems better, more in line with "best practices", etc, and we already do similar things when we "promote an Orbit build" (That is, there is an instance of Eclipse, that runs tools, etc.). 2) More important, I'd be concerned about the way your code currently "destroys the current, latest link" and then "creates new composite with the latest" (if I'm understanding your code). There is some chance that someone could be just at that point in their build where their build would break with "Orbit bundle not found" with that sort of "sudden change". Perhaps an alternative, but similar implementation (but using p2 ant tasks, etc.) would be to add "the latest" I, S, M, build, etc., at the same time it is "promoted" and then once every few days (or week?) go through and remove older versions. I think that would allow any build "in progress" to complete without error, though release engineers might still need to "react" before their next build ... at least there is a little wider window. I don't mean any of this as criticism, and still applaud your initiative, but I think to be provided by Orbit it'd have to be done with these considerations in mind. Still interested?
(In reply to David Williams from comment #12) > Thanks for the explanations. And, I really do feel like Orbit is the place > for these Yeah, that was my immediate thought, too. > [...] If you were willing to do "in Orbit", I'd like to see it done/changed as > follows: > > 1) Use p2 APIs (such as ant task, or p2 application) to create/remove things > from the composite, instead of some "manually" written XML ... not a big > deal ... just seems better, more in line with "best practices", etc, and we > already do similar things when we "promote an Orbit build" (That is, there > is an instance of Eclipse, that runs tools, etc.). As I already explained to Mikael Barbero I don't see much added value in blowing up the tiny and fast script with a full OSGi process and p2 code just to spit out five lines of XML. I assume it's highly unlikely that p2 would ever discontinue to support the current format even in the case a new format gets introduced in the future. I'm still waiting for a comment from Pascal about this. > 2) More important, I'd be concerned about the way your code currently > "destroys the current, latest link" and then "creates new composite with the > latest" (if I'm understanding your code). There is some chance that someone > could be just at that point in their build where their build would break > with "Orbit bundle not found" with that sort of "sudden change". If I'm not missing an important aspect here you're talking about the millisecond between writing the two tiny XML files, i.e., compositeContents.xml and compositeArtifacts.xml, which happens, as you say, at most once a week, sometimes, once only every few months. I don't see this as a major risk; at least not compared to what we have today, i.e., the risk that someone doesn't update his build script with the newest Orbit link and hence contributes an entirely wrong set of Orbit bundles to the train. BTW. I do think that ordinary symlinks would do the best job, but for some reason they just don't work. I couldn't figure out whether they're rejected by apache or p2. I tried to make it work with a custom .htaccess file but that was either incorrectly written or just not picked up. Maybe the apache server config doesn't allow to override it. > Perhaps an alternative, but similar implementation (but using p2 ant tasks, > etc.) would be to add "the latest" I, S, M, build, etc., at the same time it > is "promoted" and then once every few days (or week?) go through and remove > older versions. I think that would allow any build "in progress" to complete > without error, though release engineers might still need to "react" before > their next build ... at least there is a little wider window. I'm not sure what that would buy us. Are you concerned that the compositeArtifacts.xml could be read a long time after the compositeContents.xml? I think the worst thing that could happen in this case is that an IU that has been resolved against an "old" compositeContents.xml doesn't find its artifacts in the compositeArtifacts.xml later -- and fail before it could be contributed to anywhere. > I don't mean any of this as criticism, and still applaud your initiative, > but I think to be provided by Orbit it'd have to be done with these > considerations in mind. > > Still interested? No, not so much with all the extra work that you seem to ask for and that I see not much value in ;-) I do think it should be done as part of the Orbit build and not as a cron job. Maybe it is simpler there to use p2 APIs (because you already have the needed p2 installation), but I'd prefer to use my current solution rather than spending a couple days to figure that out.
(In reply to Eike Stepper from comment #13) > BTW. I do think that ordinary symlinks would do the best job, but for some > reason they just don't work. I couldn't figure out whether they're rejected > by apache or p2. I tried to make it work with a custom .htaccess file but > that was either incorrectly written or just not picked up. Maybe the apache > server config doesn't allow to override it. If you are talking about symbolic links ('ln -s') then the answer is quite simple: The web server (and especially the download server mirrors) are not following any symbolic links for security and other reasons. (Denis could give exact reasons). Hard, physical links ('ln -P') could work on the eclipse.org file system, but I doubt that the mirror servers would get these hard links by a rsync synchronisation.
(In reply to Markus Knauer from comment #14) > If you are talking about symbolic links ('ln -s') Yes. > then the answer is quite > simple: The web server (and especially the download server mirrors) are not > following any symbolic links Yes, that's what I figured. > for security and other reasons. (Denis could give exact reasons). That would be interesting. I've cc'ed Denis. > Hard, physical links ('ln -P') could work on the > eclipse.org file system, but I doubt that the mirror servers would get these > hard links by a rsync synchronisation. I've tried to create a hardlink to it doesn't seem to be possible for directories, only for regular files.
> > simple: The web server (and especially the download server mirrors) are not > > following any symbolic links > > Yes, that's what I figured. > > > for security and other reasons. (Denis could give exact reasons). > > That would be interesting. I've cc'ed Denis. Adding a link to /etc/something could be dangerous for us. > > Hard, physical links ('ln -P') could work on the > > eclipse.org file system, but I doubt that the mirror servers would get these > > hard links by a rsync synchronisation. In essence, this is the reason we don't allow symlinks or .htaccess redirects on download.eclipse.org... besides being slow, they are not guaranteed to work on mirror sites.
(In reply to Eike Stepper from comment #13) > > > > Still interested? > > No, not so much with all the extra work that you seem to ask for and that I > see not much value in ;-) > Ok, well, too bad ... I mean that as a compliment ... Orbit could use someone with your initiative. And, "thanks for trying" ... out of all the people who have complained about this over the years, you are the only one (I know of) that has tried to do something about it. While I disagree with your solution, I sincerely appreciate the suggestions and comments. Thanks again,
*** Bug 499341 has been marked as a duplicate of this bug. ***
Eike, your proposal from bug 499341 sounds like a good offer. :) FWIW, the reason I don't like making composite repositories would be because of them including *all* the old stuff. Loading composite repositories in p2 is pretty inefficient currently (separate multiple http calls for each child). I would be find with having a latest for each build type pointing just to the latest.
Doing a mass "reset to default assignee" of 21 bugs to help make clear it will (very likely) not be me working on things I had previously planned to work on. I hope this will help prevent the bugs from "getting lost" in other people's queries. Feel free to "take" a bug if appropriate.
We currently have : http://download.eclipse.org/tools/orbit/downloads/latest-N http://download.eclipse.org/tools/orbit/downloads/latest-I http://download.eclipse.org/tools/orbit/downloads/latest-S (there will be a latest-R once we release our final contribution to Oxygen release train) which are composites pointing to the latest build for that type. These are updated any time a build on the HIPP for a particular build type succeeds as opposed to a polling approach since we know exactly when a build happens. I could probably announce this in addition to making them visible somehow from the downloads page with a warning that consumers should take extra care to ensure they're getting the exact bundle(s) they want.
Thank you, that's terrific! I'm sure folks will appreciate this a lot. Please announce it publicly. BTW. I've already added the repos to my builds and they work well.
(In reply to Roland Grunberg from comment #21) > We currently have : > > http://download.eclipse.org/tools/orbit/downloads/latest-N > http://download.eclipse.org/tools/orbit/downloads/latest-I > http://download.eclipse.org/tools/orbit/downloads/latest-S > (there will be a latest-R once we release our final contribution to Oxygen > release train) > > which are composites pointing to the latest build for that type. These are > updated any time a build on the HIPP for a particular build type succeeds as > opposed to a polling approach since we know exactly when a build happens. > > I could probably announce this in addition to making them visible somehow > from the downloads page with a warning that consumers should take extra care > to ensure they're getting the exact bundle(s) they want. That's really great! I will update our builds too.
I'll mark this as CLOSED (FIXED). Possible future improvements could include falling back onto the more stable repository (eg. if choosing latest-N, and none exist, should fallback to latest-I, latest-S, latest-R).
