[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] Transparent P2 mirrors (Was: Why (not) a "Final Daze" in SimRel?)

Hi all,

I fully agree with what Carsten said, plus 2 more observations:

1. If "everybody should use p2.mirrorsUrl", does the CBI support adding this with Maven/Tycho? Are there any official docs how to persuade Tycho to add that slot? The only doc that I could find was a blog from Andreas at Codetrails, but that doesn't look like "the Eclipse.org official story for something every project should do"?

2. If I prefer downloaded Zips of p2 repos for local archiving and competely local install (since I'm behind a company proxy that only my Web Browser can bypass), I found the p2.mirrorsUrl inside the ZIPs to be not very helpful. Is it possible adding the p2.mirrorsUrl slot to a composite, instead of the content.jar of individual children? That would make more sense to me (and also alleviate part of Carsten's concerns).

Thanks,
Martin
--
Martin Oberhuber | Software Architect, Project Lead & Consultant | Austria

Am 2017-06-27 um 18:14 schrieb Carsten Reckord:
I must echo something that Ed touched on:

* Doesn't the Foundation plan to enable a transparent mirroring
system very soon that would make all this p2.mirrorsUrl useless?
No!!!  With mirror URLs, the mirrors are directly accessed with no
further access to download.eclipse.org.  With transparent mirroring,
the download server remains a bottleneck because it must be consulted
in order to redirect "transparently" to some other site.
Agreed -- transparent mirroring is nice for zip files and such, where a
short URL can be convenient. But for p2 repos which are accessed by
machines, NO!!!!  :)  Let the client gather bits from the local source
directly.
This goes waaaay off-topic from the original discussion, but:

Depending on how the transparent mirroring works, I could actually see it help a lot for P2 repos as well - at least with a little support from P2.

A) The situation right now is, for some fictional composite release update site on eclipse.org, and a very simplified P2 install:

1. Get http://download.eclipse.org/foo/updates/latest/p2.index
2. Get http://download.eclipse.org/foo/updates/latest/compositeContent.jar
3. Read child URL "../../drops/1.2.3", resolve it to http://download.eclipse.org/foo/drops/1.2.3
4. Get http://download.eclipse.org/foo/drops/1.2.3/p2.index
5. Get http://download.eclipse.org/foo/drops/1.2.3/content.jar
6. Resolve the P2 install plan
7. Get http://download.eclipse.org/foo/drops/1.2.3/artifacts.jar
8. Read p2.mirrorsUrl from artifacts.jar
9. Get mirror list from p2.mirrorsUrl (e.g. http://www.eclipse.org/downloads/download.php?file=/foo/drops/1.2.3&format=xml)
10. Select a mirror from the list and download individual artifacts from the mirror

While step 10 is typically the one that transfers the most actual bits over the pipes, that's actually six requests with non-trivial content that hit eclipse.org before we ever reach the mirror. In my experience, this already adds up to some significant delay if download.eclipse.org has a slow day or your pipe to it is thin in general. 

Also, having to mess with the artifacts.xml, e.g. to provide this repository completely from your own mirror, is quite messy. And having your Eclipse still contact download.eclipse.org due to the p2.mirrorsUrl when you've put up a mirror e.g. on a local server can be quite a surprise.

B) Now imagine a transparent mirroring solution where step 1 immediately redirects us to a suitable mirror, and P2 cooperates:

1. Request http://download.eclipse.org/foo/updates/latest/p2.index
a. Receive HTTP 307 to http://mirror.example.org/foo/updates/latest/p2.index 
b. Update P2 repo object so further repository loading uses the redirected URL
c. Follow redirect and get http://mirror.example.org/foo/updates/latest/p2.index 
2. Get http://mirror.example.org/foo/updates/latest/compositeContent.jar
3. Read child URL "../../drops/1.2.3", resolve it to http://mirror.example.org/foo/drops/1.2.3
...

That's a single request to eclipse.org (maybe two or three if there is no p2.index and P2 has to guess) and we're off to a faster mirror. And all that without messing with p2.mirrorsUrl. Also, it's much closer to how HTTP redirects are handled on the web and feels a lot more natural, to me at least.

Right now, this doesn't work because P2 transparently fetches the redirected content for each request, one at a time, but does not interpret it as a location change for the whole repo (which means you still get the first sequence of requests, just with a redirect after each step). I never quite understood the rationale behind this behavior. Was this a concious decision, and for what use-case? Or is it just coincidentally the ECF transport's redirect handling "just doing its job"?

WDYT?
Carsten
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev