[linuxtools-dev] [stephane.bouchet@xxxxxxx: Re: [cross-project-issues-de

[Andrew Overholt <overholt@xxxxxxxxxx>]

This could work for us assuming that Tycho doesn't grow the ability to
write this information itself.

----- Forwarded message from Stéphane Bouchet <stephane.bouchet@xxxxxxx> -----

> Date: Fri, 16 Mar 2012 11:46:28 +0100
> From: Stéphane Bouchet <stephane.bouchet@xxxxxxx>
> To: cross-project-issues-dev@xxxxxxxxxxx
> Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do better
> 
> Hi,
> 
> i created some ant scripts to do so using tycho based build. see [1]
> 
> however, i also discovered that earlier builds using buckminster had
> incorrectly pointing "p2.mirrorsURL" to a wrong location[2].
> i don't know if other projects are affected, but Acceleo, ATL, EMF
> Compare and EEF were impacted.
> 
> hope this will decrease the amount of p2 requests,
> 
> cheers,
> 
> Stéphane.
> 
> [1]http://sbouchet-eef.blogspot.com/2012/03/p2-repositories-we-can-do-better-eef.html
> [2]https://bugs.eclipse.org/bugs/show_bug.cgi?id=314388#c69
> 
> 
> Le 02/03/2012 07:05, David M Williams a écrit :
> >
> >
> >That is, we can do better at educating people how to make a good p2
> >repository! We have not done that education well. Hopefully this note, with
> >its links, can help provide the education you need .... then tell your
> >friends! ... tell your family! :)
> >
> >This information actually applies to all projects and all releases at
> >Eclipse, not just the yearly train, but I am sending to cross-project list,
> >thinking that will educate most projects as a first step. This should be
> >considered a requirement of correct and effective use of the Eclipse
> >Foundation Infrastructure so mentors and PMCs should make sure their
> >projects know about it and making effective use of the infrastructure.
> >
> >I think everyone knows about pack200, and while sometimes a pain, it can
> >help reduce bandwidth for faster installs, and I think most projects do a
> >good job of using that. And that is one of the hardest things to do! ...
> >which makes me think the current situation is more about education than
> >amount of effort.
> >
> >There are two things which, as a whole, we are not doing well.
> >
> >A. Use p2.mirrorsURL property in artifact repositories (artifacts.jar/xml
> >files).
> >
> >See bug 368826. There are many reasons for not using it ... such as lack of
> >tools, but another reason is that some of us (e.g. me) thought it was
> >"common knowledge" that you should use it, but of course it is not to
> >projects that are new to Eclipse. In response, I have added a little
> >documentation to the main IT Doc [2], beefed up and corrected some of the
> >community maintained page that describes how to use p2.mirrorsURL [3] and
> >provided some documentation on how to use a custom Eclipse Application to
> >set, change, or remove the p2.mirrorsURL, courtesy of the Webtools Releng
> >project [4]. While an "unofficial" tool, I think most can take advantage of
> >it, as is. Just practice on some test repos first! :) But this type of tool
> >can make it easier to fix existing artifact repositories without
> >hand-editing a 2 Megabyte XML file. But again, practice on some test repos
> >first! You can make a copy of (only) the artifacts.jar file to some /temp
> >directory your local machine and "pretend" that is the artifact repository,
> >to see if it makes the changes as you intend. Then use your scripts on
> >production machine, being sure to copy original artifacts jar to a safe
> >location first in case you need to revert, and be sure to test its really
> >working, afterwards, as is well described on the p2.mirrorsURL wiki[3].
> >
> >B. Use p2.index at repository locations.
> >
> >p2 likes metafiles for its metafiles!  As if content, artifacts,
> >composites, jars and xml were not enough, a while back p2 found the need to
> >use another file, called p2.index to help it figure out a few ambiguous
> >cases. Since few people need this functionality, it has gone ignored for a
> >long time but it is finally becoming noticeable, as Eclipse gets larger and
> >more popular (and, as a victim of its own success, as "p2 update" is
> >working well enough many people use that now, instead of downloading a
> >fresh zip file!). The "lucky accident" is that providing the p2.index file
> >can save p2 a few needless "round trips" to the download server, while it
> >figures out what is there, and what it should use, even if not truly needed
> >to resolve ambiguity. The long gory history is documented in bug 347448
> >[5]. In response, I have added a little documentation to the main IT Doc
> >[6] and beefed up and corrected the p2.index wiki [7]. The good news is
> >that it is easy to do. For 99% of you (I am guessing) it is simply a matter
> >of copying one of two files to your repository locations ... you do need to
> >pick which goes where (and most of you will need both, one at composite
> >sites, the other at simple sites). The two to pick from (for these
> >frequent, common cases) are documented on the wiki [7] (if you find other
> >common, or, even tricky cases, please add them to the wiki).  This is
> >especially important to have for "composite sites" since p2 normally checks
> >for that last, but if the p2.index says that location is a composite site,
> >then that saves p2 several needless "peek attempts" at download server.
> >Note, recent versions of b3 aggregator [8] produce these p2.index files
> >automatically, though the default Equinox p2 publisher does not (bug 302909
> >[9]) (I do not know about other p2 publishers, if there are any).
> >
> >While all these issues and p2 in general can be made better, faster,
> >easier, and some day even mow your lawn (over time, if someone commits to
> >it) we are where we are, right now. So these are two things we
> >can/should/need to do right now to improve the situation: add p2.index
> >files to repository directories, and add p2.mirrorsURL property to artifact
> >repositories.
> >
> >I know, this is a long note. And even tons more information linked to go
> >off and read for details. Keep this note for reference; bookmark the links.
> >Discuss it with your team. I suspect only one person per project needs to
> >understand it enough to act one it, but the main point is that there are
> >things that can be done right now (over the new few weeks, month) to
> >improve our p2 repositories (even old, existing ones -- the stats say even
> >helios repositories are still getting LOTS of use!) and then we would of
> >course clearly want to continue to do these good practices for Juno and
> >Kepler. Maybe one person per project could spend one day on it within the
> >next 2 to 4 weeks? (And, yes, I know that is asking for a lot of total
> >effort ... but, you will learn a lot of cool things in the process! ... and
> >cleaning up and learning these things now will help us be ready for the
> >Juno release!
> >
> >Doing these things will give our Eclipse users a little better update and
> >install experience.  To be clear and honest, this is not gong to be like a
> >50% improvement, or anything ... I suspect it won't be noticeable most of
> >the time for most users ... but, some users will notice it a lot! if they
> >can get mirrors closer to their corner of the world (via p2.mirrorsURL
> >property), and it will be noticeable by many people during those "peak
> >times" then download.eclipse.org gets flooded by huge numbers of tiny
> >requests for files that will never exist (such as "peeking" for
> >content.jar, when p2 could look more directly for compositeContent.jar, if
> >p2.index was there).  [If our webmasters didn't know about p2, they'd
> >probably think it as a DOS attack :) just kidding, I've no idea what a DOS
> >attack looks like, and am pretty sure our IT infrastructure is responding
> >well.]
> >
> >Three years ago, who would have thought p2 would be a victim of its own
> >success?! :)  I am sure it can be even more successful and more "enterprise
> >ready", so if anyone is interested, get involved! provide some patches!
> >make some tools!
> >
> >As always, feel free to ask if questions ... and feel free to correct any
> >mistakes I make in my statements or my wiki updates.
> >
> >Much thanks for your time and efforts.
> >
> >
> >p2.mirrorsURL related:
> >
> >[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=368826
> >[2]
> >http://wiki.eclipse.org/IT_Infrastructure_Doc#Enable_mirrors_.2F_use_mirrorsURL_for_my_p2_repo.3F
> >[3] http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
> >[4] http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
> >
> >p2.index related:
> >
> >[5] https://bugs.eclipse.org/bugs/show_bug.cgi?id=347448
> >[6]
> >http://wiki.eclipse.org/IT_Infrastructure_Doc#Include_a_p2.index_file_at_p2_repository_site.3F
> >[7] http://wiki.eclipse.org/Equinox/p2/p2_index
> >[8] http://wiki.eclipse.org/Eclipse_b3/aggregator/manual
> >[9] https://bugs.eclipse.org/bugs/show_bug.cgi?id=302909
> >
> >
> >_______________________________________________
> >cross-project-issues-dev mailing list
> >cross-project-issues-dev@xxxxxxxxxxx
> >https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
> >
> 

> begin:vcard
> fn;quoted-printable:St=C3=A9phane Bouchet
> n;quoted-printable:Bouchet;St=C3=A9phane
> org:Obeo
> adr;quoted-printable:BP 20773;;7 Boulevard Amp=C3=A8re;CARQUEFOU;;44481;France
> email;internet:stephane.bouchet@xxxxxxx
> tel;work:02-51-13-61-67
> x-mozilla-html:FALSE
> url:http://www.obeo.fr
> version:2.1
> end:vcard
> 

> _______________________________________________
> cross-project-issues-dev mailing list
> cross-project-issues-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev


----- End forwarded message -----