[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [p2-dev] Re: [orbit-dev] Best way to get a binary copy of the latest Orbitbundles

> > Not sure if anyone is interested but we can support building bundles
> > with Tycho and deploying the bundle to the remote Nexus.
> When would the p2 metadata generation happen?

> As the bundles are deployed.
> Also what happens to the metadata in case of a copy from repos, I
> assume that Nexus stores it too?

> If by copy you mean a physically host P2 repository, we would
> regenerate the metadata to make sure it's intact.

This is a major problem. Metadata should/can not be regenerated. The main reasons are:
1) An IU should be treated immutably. Id + version == immutable content. The IU is for a particular artifact is the responsibility of the owner of the artifact.
2) The binary can not always be reverse engineered to produce the IU that the owner of the artifact would have created. It happens that you have been successful so far because orbit metadata is a straight transcription of the bundle manifest. However this breaks when a) the generator is changed, b) the owner of the artifact used a p2.inf to specify more information (e.g. overwrite capability, requirement, etc). c) the artifact is not something that p2 metadata gen recognize.
3) Also some IUs are pure metadata, there is no artifact associated with it that needs to be stored and deployed. An example of such a thing is the top leve IU of a product, or the configuration units.

> for Maven repositories as well in Nexus. In the case of the Orbit
> zip we downloaded it didn't seem to have any so we just generated
> it. Right now we are using the standard P2 metadata generator but
> we're finding it uses a _lot_ of memory so we're not sure if it's
> something that's known and will be fixed. We might take a crack at
> writing one as while we are combining many repositories it's
> doubling the normal memory requirements of Nexus.

Most of the memory consumption is caused by the vast amount of strings that result from the generation because we do not try to intern at generation time like we do at runtime. But  the core issue is the SimpleMetadataRepository implementation that loads the complete repo. If you are really worried about memory consumption rather than changing the generator I recommend you to look at an alternative implementation of the MetadataRepository. But in reality the best solution would be to simply reuse metadata when it is provided.

[...]


> > have a "lockdown list" of sorts which is really to pick the root set
> > of IUs which basically hides any other versions for stability. A
> > profile essentially.
> In p2, a profile is the result of an installation. Do you mean that
> this is software made to work together like a "distro" (or repo
> controlled line up), e.g. all the milestones sites for a set of repos?

> Yes, it is essentially a versioned distribution. I might have 4
> versions of Subclipse in a hosted repository but I don't want the
> resolver to "see" it. Basically locking down the first level of IUs
> so that you can definitively control what the client is going to
> consume. Our use case is that for a Eclipse installation you need
> something definitive, and for server side your QA people are going
> to test a very specific set of bundles and say they are OK. You need
> to make sure that same set of bundles shows up on the server. If you
> use version ranges in your bundles to allow for dynamic updating you
> cannot point a resolver at a boundless repository. So the lock down
> lists are essentially a constraint to deal with the dichotomy of "I
> must know that what I use is what I tested, but I still must be able
> to incrementally update bundles. So this lock down (or whatever we
> end up calling it) is something that a QA engineer could manage. The
> QA person tests a new bundle and updates the lock down list and
> creates a new version which might reveal 1..N new versions of
> bundles. The client on the other side when resolving will now see
> the new versions and update. We didn't really see a large managed P2
> repository being very useful for production purposes without this
> constraining layer. The case where you are continually deploying to
> a P2 repository and updating, you definitely do not want a resolver
> picking up new versions just because they happen to be in the
> repository. We figured this provides the flexibility so that you
> have content you can actively manage as opposed to producing static
> repositories with one-time generated metadata which will soon be out
> of date. Especially in cases where you are repeatedly building new
> bundles test over and over again.

I understand the need and see where you are coming from. This is definitely valuable when you have the complete control over the set of repositories that any user can see at anytime. However, it is a bit fuzzy to me on how this will work for people who are adding random repos that they found on web. Eventually would not they be exposed to content they are not supposed to see?
At eclipse, bundle providers usually maintain several sites for different purpose (N build, M build, Release) with Galileo as the main integration hub for the community. This is our cheap version of this lockdown :) Also in the eclipse eco-system, this problem has not been encountered as much because people are still using features which provide a lock down aspect from the producer point of view thus limiting the set of choices and drastically reducing the search space. That said, we are not super happy about this and we will be looking at addressing this in 3.6.

>
> What we have is a fully working system, but we've only had it
> working for a week. But we can use this mechanism to install
> m2eclipse completely from Nexus and we are moving very fast.

Could you describe a bit more "fully working system"? From the description you've given so far, I fail to see where the configuration information is coming from and how you are integrating with p2. Do you invoke the p2 engine?

[...]

PaScaL


>  
> > Still working toward some useful and functional demos for the OSGi
> tooling summit.
> Andrew "PDE Build" Niefer and I won't be able to make it to the
> summit, but we can probably chat before that.

> I think it would be relatively straight forward to get PDE Build to
> publish and consume from Nexus if you guys wanted to try and hack it
> together at EclilpseCon.

> PaScaL
>
> > dj
> >
> > [1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=241427
> > [2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=266232
> >
> >
> >
> > orbit-dev-bounces@xxxxxxxxxxx wrote on 03/12/2009 06:30:12 AM:
> >
> > > Hi Jason,
> > >
> > > DJ is working on it:
> > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=241427
> > >
> > > Cheers,
> > > --
> > > Martin Oberhuber, Senior Member of Technical Staff, Wind River
> > > Target Management Project Lead, DSDP PMC Member
> > > http://www.eclipse.org/dsdp/tm
> > >  
> > >  
> > >
> > > > -----Original Message-----
> > > > From: orbit-dev-bounces@xxxxxxxxxxx
> > > > [mailto:orbit-dev-bounces@xxxxxxxxxxx] On Behalf Of Jason van Zyl
> > > > Sent: Donnerstag, 12. März 2009 09:25
> > > > To: Orbit Developer discussion
> > > > Subject: [orbit-dev] Best way to get a binary copy of the
> > > > latest Orbitbundles
> > > >
> > > > Hi,
> > > >
> > > > Would there be a nice way to get the latest release build of
> > > > Orbit in  
> > > > an automated way?
> > > >
> > > > Nexus is an artifact repository manager that now has
> > > > prototype support  
> > > > for proxied P2 repositories, hosted P2 repositories and arbitrary  
> > > > groupings of both types. P2 repositories in Nexus now have the same  
> > > > capabilities Maven repositories have so full RBAC access,
> > > > staging and  
> > > > promotion, grouping, ordering, routing tables, RSS feeds, audit, the  
> > > > whole nine yards. What I'm trying to do is publish as many standard  
> > > > bundles as I can in a public instance of Nexus so that users
> > > > who wish  
> > > > to try can provision bundles from Nexus in the same way a Maven user  
> > > > would provision normal JARs.
> > > >
> > > > Is there a P2 repository somewhere I can just consume what would be  
> > > > considered the latest release versions of all the bundles? Or even  
> > > > just the raw bundles are fine, Nexus can generate the necessary P2  
> > > > metadata as bundles are deployed into Nexus so I'll take anything  
> > > > really.
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > > >
> > > > ----------------------------------------------------------
> > > > Jason van Zyl
> > > > Founder,  Apache Maven
> > > > http://twitter.com/jvanzyl
> > > > ----------------------------------------------------------
> > > >
> > > > We all have problems. How we deal with them is a measure of our worth.
> > > >
> > > > -- Unknown
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > > >
> > > > ----------------------------------------------------------
> > > > Jason van Zyl
> > > > Founder,  Apache Maven
> > > > http://twitter.com/jvanzyl
> > > > ----------------------------------------------------------
> > > >
> > > > You are never dedicated to something you have complete confidence in.
> > > > No one is fanatically shouting that the sun is going to rise tomorrow.
> > > > They know it is going to rise tomorrow. When people are fanatically
> > > > dedicated to political or religious faiths or any other kind of
> > > > dogmas or goals, it's always because these dogmas or
> > > > goals are in doubt.
> > > >
> > > >    -- Robert Pirzig, Zen and the Art of Motorcycle Maintenance
> > > >
> > > > _______________________________________________
> > > > orbit-dev mailing list
> > > > orbit-dev@xxxxxxxxxxx
> > > > https://dev.eclipse.org/mailman/listinfo/orbit-dev
> > > >
> > > _______________________________________________
> > > orbit-dev mailing list
> > > orbit-dev@xxxxxxxxxxx
> > > https://dev.eclipse.org/mailman/listinfo/orbit-dev
> > _______________________________________________
> > orbit-dev mailing list
> > orbit-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/orbit-dev
> >
> > Thanks,
> >
> > Jason
> >
> > ----------------------------------------------------------
> > Jason van Zyl
> > Founder,  Apache Maven
> > http://twitter.com/jvanzyl
> > ----------------------------------------------------------
> >
> > happiness is like a butterfly: the more you chase it, the more it will
> > elude you, but if you turn your attention to other things, it will come
> > and sit softly on your shoulder ...
> >
> >  -- Thoreau
>
> > _______________________________________________
> > orbit-dev mailing list
> > orbit-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/orbit-dev

> _______________________________________________
> p2-dev mailing list
> p2-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/p2-dev

>
> Thanks,

>
> Jason

>
> ----------------------------------------------------------

> Jason van Zyl
> Founder,  Apache Maven
> http://twitter.com/jvanzyl
> ----------------------------------------------------------
>
> What matters is not ideas, but the people who have them. Good people
> can fix bad ideas, but good ideas can't save bad people.
>
>  -- Paul Graham

> _______________________________________________
> p2-dev mailing list
> p2-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/p2-dev