Community
Participate
Working Groups
Build Identifier: I20100608-0911 I don't know where i have to place this issue right, because some aspects are involved; So feel free to move to right component. The main update site for the 'Helios' release (http://download.eclipse.org/releases/helios/) is built as a p2 composite site. In opposite to a simple/normal p2 update site, a composite site contains references to aggregated sites (as a merged view). These referenced site may be also composite sites. Looking into the starting 'compositeContent.jar'/compositeContent.xml i find this content: <children size='2'> <child location='http://download.eclipse.org/technology/epp/packages/helios/'/> <child location='201006230900'/> </children> The second entry is a (good) sub directory of update site. But the first url is a mirror/update site external reference. We have a company internal eclipse mirror (via rsync from eclipse mirror service: http://www.eclipse.org/downloads/mir_request.php), but all mirrors in the world are affected by such external links, because: * Resolving url's to external references eats a lot of time. We have the mirror to get a more performant build of our eclipse based tools. * Addressing sites outside of our intranet is always a source for problems with involved proxies. * With a link like http://download.eclipse.org/technology/epp/packages/helios/ the whole mirror concept becomes worthless, because the requested content is part of the mirror, but ignored. * With that external link we lose the control over the mirror: Any artefact/version will now come into our product, because dependencies resolving cannot be controlled by hand. * We have more (expensive) traffic to the internet. So, please, please, please transform all that full qualified/external url's in the Helios update site (and in all child composite update site) into mirror interal relative url's. Reproducible: Always
This is an understandable request, but not sure how to accomplish it. Offhand, I'd think where we currently have non-relative links to repos, we'd need "up" references, such as something like ../../../technology/epp/packages/helios And, I'm not sure all mirrors have to maintain the complete 'downloads' directory structure? Do they? I think "up links" are always a little brittle, in my experience. As an alternative, perhaps instead of rsync, you could use p2 to create your internal mirror site? And, as long as you just did it once every now and then, there is a way to turn off p2's mirroring so you wouldn't have to worry about that particular mirroring pulling from elsewhere, using -Declipse.p2.mirrors=false I'm not sure what you mean by the "...cannot be controlled by hand" part of your use case, so not sure if this would help or not. Other suggestions welcome, but I think only way for us to help is for us to change our repo so everything was relative, not just change the links.
The way to create a more specific p2 mirror as simple update site we have tried in the past. Our build scripts containg the option '-Declipse.p2.mirrors=false' already, but for a composite site the option isn't working. Trying to build a update site via p2 mirroring was not successful, because to many network problems are coming up. And creating a additional p2 update site via mirror task from our local eclipse mirror is possible, but simply stupid :-) I can follow your point of view that relative paths to an artefact some steps outside of current update site directory can come to trouble... But i think this is not really a problem: public mirrors will have ALL the content, and internal mirrors (also like ours) will exclude some parts from mirroring. But thats our risk. A reference to an not existing directory is not an error/bug, simply less content in update manager to see. And the decision to exclude some directories from mirroring is mostly very sensible.
(In reply to comment #0) > * With a link like http://download.eclipse.org/technology/epp/packages/helios/ > the whole mirror concept becomes worthless, because the requested content is > part of the mirror, but ignored. I don't think so. Okay, you need one more request to get the additional content/artifact.jar's but from that point on your p2 client should choose another mirror (or your internal mirror). (In reply to comment #1) > Offhand, I'd think where we currently have non-relative links to repos, we'd > need "up" references, such as something like > > ../../../technology/epp/packages/helios Unfortunately I had to learn the hard way that this doesn't work in some circumstances. My expectation was that p2 handles these relative up-references internally, but it does not. And on the server side you cannot be sure that "/" is a path delimiter and ".." a short name for the upper level directory. In my case we tried to use such references in a AWS S3 cloud environment, the same that is used by the Eclipse Foundation as a very fast mirror. That is the reason why I would vote against a solution like this. (In reply to comment #2) > I can follow your point of view that relative paths to an artefact some steps > outside of current update site directory can come to trouble... > But i think this is not really a problem: public mirrors will have ALL the > content, and internal mirrors (also like ours) will exclude some parts from > mirroring. That's not true... all mirrors are free to decide what parts of the Eclipse download area they mirror, and in that case I know that some mirror /technology and others don't. Two other questions: * What about helios/201006230900/compositeArtifacts.jar? It references the platform artifacts with an absolute URL and I would expect that you run into similar problems with this. * What's your use case for the EPP repository content in your build? Having it available for users that use one of the EPP packages is important, but you won't find Java code in this repository that is interesting in a build.
(In reply to comment #3) > (In reply to comment #0) > > * With a link like http://download.eclipse.org/technology/epp/packages/helios/ > > the whole mirror concept becomes worthless, because the requested content is > > part of the mirror, but ignored. > > I don't think so. Okay, you need one more request to get the additional > content/artifact.jar's but from that point on your p2 client should choose > another mirror (or your internal mirror). This automatism is out of control for me. > > (In reply to comment #1) > > Offhand, I'd think where we currently have non-relative links to repos, we'd > > need "up" references, such as something like > > > > ../../../technology/epp/packages/helios > > Unfortunately I had to learn the hard way that this doesn't work in some > circumstances. My expectation was that p2 handles these relative up-references > internally, but it does not. And on the server side you cannot be sure that "/" > is a path delimiter and ".." a short name for the upper level directory. I aggree, relative path like ../ are not simple to handle. Maybe p2 will have problemd with it (thats a problem/bug for p2 not for me in this disscusion) and also security aspects (breaking out of a limited directory structure in a web server) are involved. All the arguments i heard here, were only for technical reasons. But the decision to have external references in a composite site is a conecptional issue. So we have to clarify 2 points: 1. For future (releases): Is it ok to have external links in the eclipse mirror (escpially)? 2. If 1. == no: What are pros/contras for what solutions to solve the problem (use relative references, change the whole directory structure of eclipse mirror...). There are more solutions thinkable than discussed here (as arguments gainst any change). Before we have to discuss technical problems, we need a awareness that the current design of eclipse mirror (using hadrcoded external references) is bad. > > > (In reply to comment #2) > > I can follow your point of view that relative paths to an artefact some steps > > outside of current update site directory can come to trouble... > > But i think this is not really a problem: public mirrors will have ALL the > > content, and internal mirrors (also like ours) will exclude some parts from > > mirroring. > > That's not true... all mirrors are free to decide what parts of the Eclipse > download area they mirror, and in that case I know that some mirror /technology > and others don't. Yeah, and this is one more argument *against* external references: If i use a partial (public) mirror, i don't want to get content from another host i never have declared as source for my p2 client (or build tool). If a mirror doesn't contain content i need, then i will get a error, so *i* can decide what do to (using another mirror, add a second source, mirroring more artefacts in case of internal mirror...). This has to be my decision, not one of the Eclipse foundation. > > Two other questions: > > * What about helios/201006230900/compositeArtifacts.jar? It references the > platform artifacts with an absolute URL and I would expect that you run into > similar problems with this. Mmhh, i don't fully understand the question... a) compositeArtifacts.jar and compositeContent.jar provoke the same problems. I mentioned the compositeContent.jar only to simply the disscusion base. b) A composite update site may contain other other composite update sites as children. This issue is not only touching the composite*.jar in root directory of Helios update site. > > * What's your use case for the EPP repository content in your build? Having it > available for users that use one of the EPP packages is important, but you > won't find Java code in this repository that is interesting in a build. We doesn't need anything from EPP, but the presence of a external url to the EPP on eclipse.download.org slow down our build process. And (from my original post): with these external references we have to configure a proxy, because now we must have access to the internet. The EPP reference is only a example, there are a lot of more external references in nested composite site used as child of the helios site.
I'm going to be blunt (and go out on a limb) and say we "won't fix" this. It is working as intended, and there might be ways that would be more convenient for you ... but not sure they are correct, or necessary. It does sound like you might be experiencing other bugs that deserve attention (e.g. comment #2, "Our build scripts containg the option '-Declipse.p2.mirrors=false' already, but for a composite site the option isn't working." You may a want to open a new bug in p2 component for that? (and any other issues). Finally, You say "...creating a additional p2 update site via mirror task from our local eclipse mirror is possible, but simply stupid ..." but I'm not so sure. Rsync works _now_ but part of the theory behind p2 (as I understand it) is that the repo could be implemented in other technologies besides a file system ... using a database, for example? So, using p2 APIs is definitely the right way to access and manipulate p2 repositories ... not file IO directly. Hope this helps ... and I am sympathetic ... but I don't see this getting changed so wanted to be upfront and honest about it, instead of letting it dangle. So please open bugs on specific p2 issues you find ... it can improve with your help. Thanks,
*** Bug 350031 has been marked as a duplicate of this bug. ***
