Community
Participate
Working Groups
Currently, projects use lots of different URLs for their project update sites: I'll attach a screen shot from Galilio SR1 (after the site had been contacted once, to retrieve content metadata. To give an idea of the variety, here's the ending segment of a few updates releases e3.4 update-site 3.5 2.5 1.0.1/stable milestone project=search#search Some of these look like project specific errors, but the point of this cross-project bug is that there is no common pattern. So, first of all, it looks pretty unprofessional, and hardly "coordinated". Second, there's an issue of if it _should_ be version specific or if there should just be one URL per project, and that one URL give access to any available updates? I'm not sure of the right answer, but wanted to open this bug for comment and suggestions so we can do better for Helios.
Created attachment 149010 [details] list of sites from Java EE IDE Galileo SR1
> project=search#search That's a bug in EMF Search's feature.xml. We've been asking for over a year now that they fix that but they can't be arsed, apparently. Even PMC escalation has yielded SFA. > (others) Since these URLs are defined by features, should it not be up to the feature owner to decide if the URL needs to be version-specific? Won't they be the best person to know if they feature will run on multiple Eclipses or on only a single one?
(In reply to comment #2) > > Since these URLs are defined by features, should it not be up to the feature > owner to decide if the URL needs to be version-specific? Won't they be the best > person to know if they feature will run on multiple Eclipses or on only a > single one? Mostly, but I think still best to be consistent. If possible. At least to state the principles and overall recommendations. Part of the issue is "limit of liability", so to speak. As an example, if a project uses /updates every release ... they are implying that users should expect to 'update' from one yearly release to the next. Are "we" committing to that? That doesn't work unless everyone does it. On the other hand, many do want the ability to update from one yearly release to the next, which implies a common, persistent URL should be used. Even then, projects might choose to "categorize" their features based on yearly version (and some already do). It makes my head spin thinking of all the permutations, so I'm sure it's confusing to our users as well.
Getting rid of confusion is a good reason to unify the naming. I have encountered many users thinking that a site named 'updates' is where you post updates to things that has been published earlier on a 'releases' site. I don't think that applies. Since everything is 'updates' perhaps that term should be avoided (we are guilty of using it ourselves) in favor of something that is more descriptive and says something about the state of the repository. Using one URL per project has some implications: 1. Not all features and bundles will survive mixing versions between say, 3.5 and Helios. The version ranges are just not restrictive enough. This increases the risk for corrupted installs and/or version conflicts preventing the install. 2. The concept of composite repositories didn't exist in 3.4. So obviously, it would be hard to maintain such a repository for older sites. 3. There is currently no way to say "This feature is not suitable for this purpose". We have several, headless-only, features that we don't want people to install in the IDE. Ideally, they should not show up in the "Install new software" wizard at all. For this reason we keep them in a separate update site. 4. Large repositories slow down installation. Mixing everything into one repository forces p2 to load more meta-data. That means larger downloads and a longer "Computing prerequisites" phase. 5. Features reach different levels of maturity. We must be able to make software available for download and beta-testing without including it in what we consider released material. I think that keeping a repository consistent (everything in it can be installed into the same runtime) and targeted (specific purpose, specific API baseline) is good. There's not much gain in mixing all features into one repository. Not mixing is probably a good argument for unifying naming.
See also bug 303583 for discussion of release-to-release updates. I think we should all list sites that are "release specific" (for a given yearly release), such as /webtools/updates/3.2 instead of the /webtools/updates we currently use. I'm less sure if it should be "updates" or "releases". (We in webtools might need to change to 'releases', just because we already have the plain /webtools/updates out there in the field).
(In reply to comment #5) > I'm less sure if it should be "updates" or "releases". (We in webtools might > need to change to 'releases', just because we already have the plain > /webtools/updates out there in the field). We've been generally moving away from using the term "updates". Generally this terminology dates from the era when the main thing an "update site" could be used for was updating an already installed application. Repositories today can be used for many things: installing products from scratch, updating/extending products, as an input to build processes, provisioning target platforms, etc. All that to say that I think "releases" is the better term. However I understand it's hard to change these things, and the Eclipse TLP is still guilty of using the term "updates" in its repository locations. I'll have to remember to revisit that when we start on the next release (I think changing our repository locations at this point in the cycle would be too disruptive considering the marginal benefit).
(In reply to comment #6) > (In reply to comment #5) > > We've been generally moving away from using the term "updates". Generally this > terminology dates from the era when the main thing an "update site" could be > used for was updating an already installed application. Repositories today can > be used for many things: installing products from scratch, updating/extending > products, as an input to build processes, provisioning target platforms, etc. That reminds me, perhaps "repo" would be nifty? As in /webtools/repo/3.2. Or, I guess 'repository' for those new to Eclipse (best not to use abbreviations).
(In reply to comment #7) > That reminds me, perhaps "repo" would be nifty? As in /webtools/repo/3.2. > Or, I guess 'repository' for those new to Eclipse (best not to use > abbreviations). +1 for using the term 'repository' instead of 'updates'. Much clearer.
(In reply to comment #8) > (In reply to comment #7) > > That reminds me, perhaps "repo" would be nifty? As in /webtools/repo/3.2. > > Or, I guess 'repository' for those new to Eclipse (best not to use > > abbreviations). Rather than numbering, why not use release-train names? /webtools/repo/galileo /webtools/repo/helios /webtools/repo/adnauseum ... That way there's no confusion re: what version of WTP matches to what version of Eclipse -- you just look for all the URLs w/ matching release train name suffix, and bob's your uncle (fanny's your aunt) you know what to install. > +1 for using the term 'repository' instead of 'updates'. Much clearer. As to "repo" vs. "updates"... same crap, different pile. I get updates from a repo; my repos contain updates. I like repo because it's shorter, however, and unless people start using "repo" and "repos" and "repository" variants, it gets us away from "update" vs. "updates" vs. "update-site". #imjustsayin'
> > Rather than numbering, why not use release-train names? > > /webtools/repo/galileo > /webtools/repo/helios > /webtools/repo/adnauseum > ... > I like this idea. The "marketing number" doesn't do anyone much good anymore ... since its not used in the software ... especially as more and more features within a site diverge from a common number. Not sure what non-release train projects should do ... there is some sense in them using the the release train name, if it means "goes with" this release train ... but I sort of think the release train name should be reserved for those on the release train. I guess they'd still use a "marketing number". What about next level down, in composite repositories? Any one have strong preferences? I prefer "timestamps" myself, but know others would prefer "SR0", "SR1", "SR2". Not that these would ever be used in a feature update URL ... just wondering if there's anything we should recommend? I like the timestamp directories, since it more naturally accommodates off-cycle releases. I guess the other variable is "sub project". I'm sure this would vary from Top Level Project to Top Level Project, but is there ever a need to have sub-branches within a TLP? Such as /webtools/jsdt/helios? I pretty sure not for webtools ... less sure about that complicated Modeling Project :) Again, I'm just thinking of what to recommend to people, if they ask, not what to dictate to anyone.
(In reply to comment #10) > What about next level down, in composite repositories? Any one have strong > preferences? I prefer "timestamps" myself, but know others would prefer "SR0" I'm slightly confused... are you saying that in order to properly support the release train's composite repositories, each project needs to change their update site organization to include timestamps or service release folders? IMO one thing not to forget is that repositories should still be used to easily "update". In order to make that possible, the base release (e.g. eclipse 3.5) needs to know the location where to look for updates of itself. Therefore, any "service release" updates must at least be accessible from the base update site. I do not think that updating everything via the composite train site will work, since many projects provide part-features (incubation etc) that is supposed to be udpateable, but not listed in the train repository.
> > I'm slightly confused... are you saying that in order to properly support the > release train's composite repositories, each project needs to change their > update site organization to include timestamps or service release folders? > They would not literally have to, but its a pattern I'd recommend. I'm not sure everyone would recommend the pattern, but it has advantages. But, to be explicit, let me give a detailed example, and perhaps that will help uncover if there's confusion in what's meant ... or, disagreement about best way to do it. I am learning that it's important that once something is available to be installed from a repository, it should always be available to be installed from that repository. (Good so far? Or is that a point of contention?) So, let's say webtools repository for helios is created, and lets say it has 10 features and 100 bundles. We'll call that "SR0" meaning the first release. Later, SR1 comes out, and let's say it has 5 features updated, and that 50 of the bundles change, and 50 stay the same. alternatives: (In both cases the only URL that's "surfaced" to the user is always, only /webtools/repository/helios). Alternative one: Literally only one (non-composite) repository is used. So under /webtools/repository/helios you end up with 15 features and 150 bundles. All directly under that 'helios' directory, with a features directory, plugins directory, a content.jar and an artifacts.jar Alternative two: SR0 is put in /webtools/repository/helios/sr0 and SR1 is put in /webtools/repository/helios/sr1, so "grand total, you'd have some duplication, with grand total of 20 features, and 200 bundles. Each srN diretory would have its own features and plugins directory and content.jar and artifacts.jar, and /webtools/repository/helios/ would have, in addition to the sr0 and sr1 directories, compositeArtifacts.jar/xml and compositeContents.jar/xml. The advantage of alternative 1 is that it is smaller (taking less server space). There are a few advantages to the second approach, though. 1. It is more forgiving of mistakes. If, for example, you discover you did something wrong in creating sr1, its relatively easy to re-create it, remove the old one with file/shell commands. If you "merge" them, then you have to undo that merge, somehow, before re-merging with the new sr1. 2. It is easy to "rollout". You can created and provide the 'sr1' directory, and let its artifacts mirror for a few days before you actually make it "visible" to the composite repository at /webtools/repository/helios; once you do make it visible, then everything is ready to go, without all artifacts trying to come from Eclipse.org. 3. As time goes on, and let's say there's an sr2, sr3, sr4 after a few years (for sake of illustration). At some point you can decide to move sr0 and sr1 artifacts to a non-mirrored location, 'archive.eclipse.org', relatively easily (that is, with a few shell file command, and a few one-line changes to composite jars/xml files. 4. Having the hierarchical structure gives you the most flexibility for performance advantages, if required. As above, let's say you have several service releases, and you end up with thousands of features and bundles. If you have only the merged repo, then you are pretty much locked in to that. If you have the (redundant) hierarchical structure, you would at least have an easy fallback of directory users (or builds) to use a more specific URL, such as /webtools/repository/helios/sr4 to get that content in a faster way. Similarly, there might be cases where you'd want to direct people, for special purposes, to use only the highest level repo, at /webtools/repostory, and you could get stuff from many different release streams. So, that'd be my advise. Does that help reduce the confusion? ... or add to it? :) We are getting fairly detailed for the original purpose of this bug, which was to avoid the wild variety in update site URLs ... but I think worth it, since I suspect some people would like hearing "how to ..." when they hear about what names they should use. Hopefully we can converge on a good set of advice. (BTW, you are correct in some of your last comments, and I agree release train name could not always be used, if there were 'incubator' repositories, etc.)
(In reply to comment #12) Thanks for the detailed explanation! I see how all you are saying makes sense (and how it could have relieved some of my pain with "merging" sites in the past). Actually one problem I'm seeing today with having "SR0" and "SR1" in one repository is that even though I list all SR1 features in a separate SR1 category, users of the repository typically only see a subset of the SR1 features in the SR1 category ... those which remain identical between SR0 and SR1 are visible in the SR0 category, and that's been confusing people in the past who wanted to "install all of SR1". Now just for the un-initiated of us... are there instructions anywhere how a composite repo can be created out of two separate ones?
This is a great initiative. To follow on the habits of eclipse, it would be good to come up with a list of must and should dos applicable to every project. On the MUST I would only put naming and retention policy a) Use repo or repos. This is shorter. b) Use the name of the targeted release train in the segment. I would use this to convention even for projects that are not part of the release train. This would go a great length to help the consumption and make it easy for a user to get new content for a random project without even googling. For example I could just type in <projectName>/repo/<releaseName> and it is very easy. c) Define repo naming convention for M, I and N builds d) Define retention policies for each repository. On the SHOULD I would put the organization of subrepos. I think the proposed organization with SR* make a lot of sense, but I don't think we should / can enforce this, since this is not necessarily suitable for all the projects. For example, there are some projects that allow for their plug-ins to be installed on both Galileo and Helios at the same time (e.g. egit).
Do we get any further with this? Are the different proposals made here discussed by the adequate councils already? I had an additional idea: Eclipse and the webmasters do an excellent job in providing us with SCM services (CVS, SVN, Git) and CI services (Hudson). It would be so cool if there was a central promotion service. A single cron job that projects can register with in order to get their builds promoted to download.eclipse.org. This job could a) encourage regular promotions like weekly I-builds b) help with/enforce the naming and layout policy c) provide a central web page that offers the promoted builds to users d) provide download web pages per project e) contribute to the release train more frequently (possibly excluding the hot milestone/RC weeks) Step by step we could enrich this service with add-on values like automatic release notes generation, bugzilla traceability, help center contribution, subscribable consumer notification, releng feedback forms, etc.
Another useful service would be the generation of human readable content of the repositories. Similar to what our CDO build generates: http://download.eclipse.org/modeling/emf/cdo/updates/4.0/4.0-M5-S201102010503 (may even be nicer ;-) )
Adding "nobody" as assignee, since I'm not actively working on this. I still think getting everyone to be uniform is too hard, and better to have a "service", something like the downloads.php page, so that if some were to ask, say for http:/repos.eclipse/org?emf;type=I then they'd get redirected to emf's I-build repo listed in their metadata, etc. ... similar for other projects. No idea of complications or even if p2 would handle redirection ... but that sort of solution would be my first choice, rather than trying to get everyone to change the way they've been doing it for years.
(In reply to comment #17) > Adding "nobody" as assignee, since I'm not actively working on this. > > I still think getting everyone to be uniform is too hard, and better to have > a "service", something like the downloads.php page, so that if some were to > ask, say for > > http:/repos.eclipse/org?emf;type=I > then they'd get redirected to emf's I-build repo listed in their metadata, > etc. ... similar for other projects. > > No idea of complications or even if p2 would handle redirection ... but that > sort of solution would be my first choice, rather than trying to get > everyone to change the way they've been doing it for years. That's a good idea. I'm thinking to have for each release declared in the projet metadata some entries for each build types [1]. This can also be usefull for people/builds to have this information.. Of course, Projects should also provide composite repos for the latest releases if they want. [1] http://download.eclipse.org/eclipse/downloads/build_types.html
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
While no one is actively working on this, it is still a problem for Eclipse Foundation as a whole ... so, removing 'stalebug' from whiteboard.
This problem got a lot worse with the introduction of even more frequent Eclipse releases (which are great!). How are users supposed to know which "random" numbers to put into the repository URLs of plugins they use? Most other software provides a mechanism to use variables in the URLs so that a release version change can be done by just updating the variable and then doing a normal update. Example: http://download.eclipse.org/releases/$releasever/ http://download.eclipse.org/eclipse/updates/$releasever/ http://download.eclipse.org/buildship/updates/$releasever/ No go figure out what you need to put into those 3 - I suspect - VERY common URLs... :P When you're done being annoyed at the inconsistency fix this issue pls.
This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/125.
