Re: [p2-dev] Subsystem provisioning

["Todorova, Katya" <katya.todorova@xxxxxxx>]

Hi,

Thanks for sharing your thoughts on that.
You can find comments inlined.

Kind regards,
Katya

> -----Original Message-----
> From: p2-dev-bounces@xxxxxxxxxxx
> [mailto:p2-dev-bounces@xxxxxxxxxxx] On Behalf Of Pascal Rapicault
> Sent: Friday, February 04, 2011 5:49 AM
> To: P2 developer discussions
> Subject: Re: [p2-dev] Subsystem provisioning
>
> I have been putting some thoughts on the issue and here is
> where I'm at atm.
>
> 1) How many profiles?
> Since we are thinking in terms of sub-systems that can have
> conflicting requirements and are logically isolated, at this
> point I think that we should probably go with as many
> profiles as there are subsystems to be provisioned.
> I have been looking at having one profile, however I'm not
> really in favour of promoting a concept similar to resolver
> hooks in the p2 solver. This would go against all the efforts
> around providing a complete solver. A possible trick to only
> run with one profile would be to have special IUs that
> under-specify the dependencies to avoid the conflicts, but
> however I don't like that because this means that there would
> be proliferation of IUs each specific to a deployment context.
>
> One thing to note though is that by using a solution that is
> based on multiple profiles, p2 may not be able to completely
> accommodate the craziness supported by the hooks. However
> this is still to be seen.

Profile per subsystem sounds good. As for the hooks I don't think that p2 should accommodate much.
In the end no matter how hooks are configured it would end up at runtime in a given set of bundles and
packages (capabilities) visible by the corresponding subsystem. This set should match to the content
of the p2 profile for that subsystem, that's all.


> 1.1) How do we represent inter profile relationships

In both cases p2 should be aware that there is a relationship among them in order to update them properly.
Change in one subsystem should trigger update of all subsystems which depend on it.

> At this point I can think of two ways to do this.
> a) One possible solution is to have one or multiple IUs
> representing what is expected from other subsystems. For
> example, in the virgo case, if the user region needed osgi
> from the kernel region (region == subsystem), then there
> would be an IU  (containing metadata only) that would provide
> the necessary capabilities. This has the advantage that the
> user can actually really express its expectations from one
> subsystem or another (this is for example what we do with the
> a.jre.iu that represents the JRE). However making sure that
> actual subsystem being depended upon is actually providing
> the appropriate functionalities is not covered.

Suppose we update the "provider" subsystem and the "consumer" update fails. Consumer's profile wouldn't be consistent
(the packages exported by the provider would be wrong), but it at least would be non conflicting.
I guess it would be easier to restore the system in that case and it's a great advantage from usability point of view.

> b) The second possibility consists in having a profile inject
> capabilities into another profile. For exemple, in the virgo
> case previously mentioned, when the user region subsystem is
> provisioned some capabilities from the kernel profile would
> be "injected" thus representing what can be done.
> The ability to "inject" capabilities could also be used to
> more adequately represent what is provided by the environment
> (for example the JRE IU could be removed to the advantage of
> such a solution).

Again comment on "failed update" scenario. Here the consumer profile would be completely broken
without a chance to be restored.

> At this point I'm not set on one solution. The  first one can
> be done today, the second would require changes in core of p2
> and would raise additional question (how are the capabilities
> to inject identified), but could be helpful in other
> scenarios. That said going down the path of the first
> solution is not irreversible.

Both solutions have advantages and drawbacks - In my opinion if there are real requirements from other scenarios
the second possibility becomes preferable and should be considered. Otherwise I would go for the first solution since it's simpler.

> 2) How many bundles.info
> Side discussion: do we want to keep bundles.info as the file
> format for describing the bundles from these subsystems? Is
> it the most appropriate format?
>
> At this point, we like to think of one bundles.info per
> profile, the bundles.info being the result of a "projection"
> from the profile into OSGi. However nothing is really
> enforced in the touchpoint to prevent several profiles into
> one bundles.info if all the profiles were set to point at the
> same locations on disk.
>
> That said, I'm not necessarily a fan of merging the files
> together for clarify purpose. In a sense it is easy to
> verify/explain that all the bundles from a profile are
> included into one bundles.info rather than having to look at
> the intersections. On top of that with the current
> bundles.info format it is possible to run into problematic
> situations where the same bundle is added from two different
> profiles but result in one entry in  the bundles.info file;
> and thus removing the bundle from one profile would result in
> removing the entry from the bundles.info thus resulting in a
> broken system.
> The other thing with one bundles.info is that I'm not sure to
> understand how the update of one subsystem can be isolated
> from the other pieces of the system.
>
> Though I understand that hooks provide a soft partitioning
> and that upon restart all the bundles are loaded from the fwk
> state at once, and thus updated in one shot, I think it is
> still desirable for clarity to keep multiple bundles.info. To
> me the key here is to not use the normal simpleconfigurator
> (or enhance it) but have something that upon server restart
> will read all the bundles.info and apply them all in one shot.

Currently there is no relationship between bundles.info and profile and people are used to specify just a profile and have their stuff properly installed.
So on one hand having multiple bundles.info would make the end user unhappy. On the other hand it would be hard to keep a mapping between bundles.info and profile files.
On top of that all bundles.infos should be listed somewhere to be sure that the installed system is complete. It's too complex and I don't think it is necessary.
Instead, we can use an idea from hooks specification - it says that two "instances" of one bundle could be installed in different subsystems if only they are placed at different locations.
In bundles.info there is already a record of bundle location so no need to change the file format. Also it would be easy to see where each bundle belongs because of the bundle location.
That's how I see subsystem isolation in bundles.info.

The only change which should be made is using different locations for storage (relative to the end system destination).
The location could be a property in the corresponding profile of the subsystem.


> 3) How are the hooks configured
> As for the configuration of the hook, it would be in my
> tendency to have this be externalized (think hooks.info) and
> be controlled through metadata (in this case touchpoint
> actions).  I like things to be explicit, maybe sometimes to
> the detriment of concision, because I believe it helps when
> trying to figure out why the system is behaving the way it does.
> On top of that for the application developer (or someone
> developing an app fwk to be itself run in isolation, but also
> the kernel developer) it can be important to be able to
> clearly specify in some form or another how the hooks for its
> part are being configured.
> Here again, we may be limiting what can be done with the
> hooks by defining a bunch of directives in a file, but I
> don't think it is a bad thing nor that we will be running
> into many issues. Somehow I have the feeling that it could be
> enough to have information in the file that represent what is
> imported / exported by the subsystem in a style similar to
> what a bundle does. Indeed maybe this hooks.info is not one
> file but a set of manifest.mf files.

I agree that having hooks information explicit is very useful because of the reasons you mentioned.
There is just one drawback I can think of - since this information is contained in the profile as well there should be a synchronization step in order to prevent inconsistency.

> Finally to conclude, it needs to be noted that I have not
> coded anything mentioned in there, but it worth noting that
> there is in fact 3 problems that can be addressed separately
> but also that much of today's p2 can actually be reused or
> enhanced and that we are not that far.
>
> The question left open now is how to provision in one shot
> (or from one IU?) several subsystems. I have been looking at
> some side metadata that would partition an existing
> dependency tree, however I don't have concrete to propose yet.
>
>
>
> On 2011-01-31, at 8:49 AM, Todorova, Katya wrote:
>
> > Hi,
> >
> > You can find answers inlined.
> > I hope this conversation will help us gather a consistent
> list of requirements and will let us come up with a
> reasonable concept covering that topic.
> >
> > Thanks for you comments,
> > Katya
> >
> >> -----Original Message-----
> >> From: p2-dev-bounces@xxxxxxxxxxx
> >> [mailto:p2-dev-bounces@xxxxxxxxxxx] On Behalf Of Pascal Rapicault
> >> Sent: Monday, January 24, 2011 9:18 PM
> >> To: P2 developer discussions
> >> Subject: Re: [p2-dev] Subsystem provisioning
> >>
> >> I'm not completely familiar with Virgo or the fwk hooks, but
> >> the first thing that comes to mind when I hear subsystem is
> >> "isolation" and "conflicting dependencies", in the sense that
> >> each subsystem is relatively isolated from each other, and
> >> can therefore have dependencies that are conflicting with
> >> each others (e.g. two different version of the same
> >> singletons). Is this correct?
> >
> > Yes, subsystems are about isolation at runtime - they can
> be considered as different systems which are able to
> communicate to each other by sharing resources (packages,
> services, properties, etc).
> > So the problem with conflicting dependencies when you
> install two different versions of singletons in one OSGi
> framework wouldn't exist if you install them in two
> subsystems. (Of course, it depends
> > on how you define the visibility rules)
> >
> >> Consequently, when I look at this from a p2 standpoint, the
> >> first question that comes to mind is how are we going to do
> >> dependency resolution? Also in a more advanced scenario how
> >> do the two subsystem evolve?
> >
> > The dependency resolution mechanism should be aware somehow
> that installed artifacts are part of different subsystems and
> resolve against corresponding framework state. Currently in
> p2 there is
> > one profile per installation, so a naive solution could be
> introducing a flag in the profile format and filter its
> content depending on that flag during resolution. Or for
> example sub profiles could be
> > created to reflect existing subsystems. A lot of questions
> pop up - how would you trigger installation in a particular
> subsystem, would the profiles be isolated or would they share
> some pieces of information, etc.
> > There are many possible solutions, all with their benefits
> and drawbacks. The challenge is to find the one which best
> fits in the p2 architecture.
> > What do you think, from p2 point of view, what means "isolation"?
> >
> >> The second question that I have is more closely related to
> >> the runtime in the sense that how does the framework know
> >> what to put in each "subsystem" ? Does each subsystem need to
> >> be in its own bundles.info file, or should they actually be
> >> merged in one? Where does the description on how to configure
> >> come from?
> >
> > As far as I know the decision what to put in different
> subsystem (or what part of the framework is seen by each of
> the subsystems) is responsibility of the system "owner" and
> could be defined via
> > configuration or programmatically. There are several open
> questions here: when do these visibility rules become active,
> could they change at runtime and what they have to do with p2
> profiles (which are the ground for p2 resolving)
> > I don't think we have strict requirements how many
> bundles.info file we have as long as the OSGi framework is
> presented in a consistent way. In my opinion, it's easy and
> straightforward to map
> > one bundles.info to one OSGi framework and leave the hooks
> to deal with visibility (resolving) at runtime. In that sense
> subsystems and p2 will complement one another without any conflicts.
> > Of course this is just one piece of the problem and should
> be considered in the whole context of profile isolation.
> >
> >> Finally what is interesting about this usecase is that,
> >> though the runtime aspect seems to relate to OSGi and fwk
> >> hooks, the more general usecase goes around how to describe /
> >> provision multiple applications (e.g. if I wanted to install
> >> m2e and maven in one shot, or if I wanted to install all of
> >> RAD that is composed of multiple small apps at once).
> >
> > While the upper questions address the installation part, I
> think here we talk about publishing and associating an
> adequate meta information with the IUs.  Since both p2
> repository and profile are used for
> > dependency resolution they need to be in sync of what a
> subsystem is. No idea how the magic metadata looks like though:)
> >
> >> All that said, though these scenarios have been discussed in
> >> the past, I don't have a solution in mind :)
> >>
> >> On 2011-01-24, at 8:38 AM, Todorova, Katya wrote:
> >>
> >>> Hi,
> >>>
> >>> As subsystems are getting more and more used nowadays, here
> >> comes the logical question how to provision them in a
> standard way.
> >>> Eclipse Virgo using subsystems to model its runtime is one
> >> example. Currently, it is very hard to install and even
> >> harder to update a particular Virgo subsystem with
> existing p2 tools.
> >>> Subsystem concept (rfc152) is not supposed to define
> >> initial provisioning process. Do you think p2 should be aware
> >> of subsystem existence and could fill that "gap"?
> >>>
> >>> Thanks in advance for your comments,
> >>> Katya
> >>>
> >>> _______________________________________________
> >>> p2-dev mailing list
> >>> p2-dev@xxxxxxxxxxx
> >>> https://dev.eclipse.org/mailman/listinfo/p2-dev
> >>
> >> _______________________________________________
> >> p2-dev mailing list
> >> p2-dev@xxxxxxxxxxx
> >> https://dev.eclipse.org/mailman/listinfo/p2-dev
> >>
> > _______________________________________________
> > p2-dev mailing list
> > p2-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/p2-dev
>
> _______________________________________________
> p2-dev mailing list
> p2-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/p2-dev
>