Re: [p2-dev] Subsystem provisioning

[Pascal Rapicault <pascal@xxxxxxxxxxxx>]

I have been putting some thoughts on the issue and here is where I'm at atm.

1) How many profiles?
Since we are thinking in terms of sub-systems that can have conflicting requirements and are logically isolated, at this point I think that we should probably go with as many profiles as there are subsystems to be provisioned.
I have been looking at having one profile, however I'm not really in favour of promoting a concept similar to resolver hooks in the p2 solver. This would go against all the efforts around providing a complete solver. A possible trick to only run with one profile would be to have special IUs that under-specify the dependencies to avoid the conflicts, but however I don't like that because this means that there would be proliferation of IUs each specific to a deployment context.

One thing to note though is that by using a solution that is based on multiple profiles, p2 may not be able to completely accommodate the craziness supported by the hooks. However this is still to be seen.

1.1) How do we represent inter profile relationships
At this point I can think of two ways to do this.
a) One possible solution is to have one or multiple IUs representing what is expected from other subsystems. For example, in the virgo case, if the user region needed osgi from the kernel region (region == subsystem), then there would be an IU  (containing metadata only) that would provide the necessary capabilities. This has the advantage that the user can actually really express its expectations from one subsystem or another (this is for example what we do with the a.jre.iu that represents the JRE). However making sure that actual subsystem being depended upon is actually providing the appropriate functionalities is not covered.

b) The second possibility consists in having a profile inject capabilities into another profile. For exemple, in the virgo case previously mentioned, when the user region subsystem is provisioned some capabilities from the kernel profile would be "injected" thus representing what can be done. 
The ability to "inject" capabilities could also be used to more adequately represent what is provided by the environment (for example the JRE IU could be removed to the advantage of such a solution).

At this point I'm not set on one solution. The first one can be done today, the second would require changes in core of p2 and would raise additional question (how are the capabilities to inject identified), but could be helpful in other scenarios. That said going down the path of the first solution is not irreversible.

2) How many bundles.info
Side discussion: do we want to keep bundles.info as the file format for describing the bundles from these subsystems? Is it the most appropriate format?

At this point, we like to think of one bundles.info per profile, the bundles.info being the result of a "projection" from the profile into OSGi. However nothing is really enforced in the touchpoint to prevent several profiles into one bundles.info if all the profiles were set to point at the same locations on disk.

That said, I'm not necessarily a fan of merging the files together for clarify purpose. In a sense it is easy to verify/explain that all the bundles from a profile are included into one bundles.info rather than having to look at the intersections. On top of that with the current bundles.info format it is possible to run into problematic situations where the same bundle is added from two different profiles but result in one entry in  the bundles.info file; and thus removing the bundle from one profile would result in removing the entry from the bundles.info thus resulting in a broken system.
The other thing with one bundles.info is that I'm not sure to understand how the update of one subsystem can be isolated from the other pieces of the system.

Though I understand that hooks provide a soft partitioning and that upon restart all the bundles are loaded from the fwk state at once, and thus updated in one shot, I think it is still desirable for clarity to keep multiple bundles.info. To me the key here is to not use the normal simpleconfigurator (or enhance it) but have something that upon server restart will read all the bundles.info and apply them all in one shot. 


3) How are the hooks configured
As for the configuration of the hook, it would be in my tendency to have this be externalized (think hooks.info) and be controlled through metadata (in this case touchpoint actions).  I like things to be explicit, maybe sometimes to the detriment of concision, because I believe it helps when trying to figure out why the system is behaving the way it does.
On top of that for the application developer (or someone developing an app fwk to be itself run in isolation, but also the kernel developer) it can be important to be able to clearly specify in some form or another how the hooks for its part are being configured.
Here again, we may be limiting what can be done with the hooks by defining a bunch of directives in a file, but I don't think it is a bad thing nor that we will be running into many issues. Somehow I have the feeling that it could be enough to have information in the file that represent what is imported / exported by the subsystem in a style similar to what a bundle does. Indeed maybe this hooks.info is not one file but a set of manifest.mf files.


Finally to conclude, it needs to be noted that I have not coded anything mentioned in there, but it worth noting that there is in fact 3 problems that can be addressed separately but also that much of today's p2 can actually be reused or enhanced and that we are not that far.

The question left open now is how to provision in one shot (or from one IU?) several subsystems. I have been looking at some side metadata that would partition an existing dependency tree, however I don't have concrete to propose yet. 



On 2011-01-31, at 8:49 AM, Todorova, Katya wrote:

> Hi, 
> 
> You can find answers inlined. 
> I hope this conversation will help us gather a consistent list of requirements and will let us come up with a reasonable concept covering that topic. 
> 
> Thanks for you comments, 
> Katya 
> 
>> -----Original Message-----
>> From: p2-dev-bounces@xxxxxxxxxxx 
>> [mailto:p2-dev-bounces@xxxxxxxxxxx] On Behalf Of Pascal Rapicault
>> Sent: Monday, January 24, 2011 9:18 PM
>> To: P2 developer discussions
>> Subject: Re: [p2-dev] Subsystem provisioning
>> 
>> I'm not completely familiar with Virgo or the fwk hooks, but 
>> the first thing that comes to mind when I hear subsystem is 
>> "isolation" and "conflicting dependencies", in the sense that 
>> each subsystem is relatively isolated from each other, and 
>> can therefore have dependencies that are conflicting with 
>> each others (e.g. two different version of the same 
>> singletons). Is this correct?
> 
> Yes, subsystems are about isolation at runtime - they can be considered as different systems which are able to communicate to each other by sharing resources (packages, services, properties, etc). 
> So the problem with conflicting dependencies when you install two different versions of singletons in one OSGi framework wouldn't exist if you install them in two subsystems. (Of course, it depends 
> on how you define the visibility rules) 
> 
>> Consequently, when I look at this from a p2 standpoint, the 
>> first question that comes to mind is how are we going to do 
>> dependency resolution? Also in a more advanced scenario how 
>> do the two subsystem evolve?
> 
> The dependency resolution mechanism should be aware somehow that installed artifacts are part of different subsystems and resolve against corresponding framework state. Currently in p2 there is 
> one profile per installation, so a naive solution could be introducing a flag in the profile format and filter its content depending on that flag during resolution. Or for example sub profiles could be 
> created to reflect existing subsystems. A lot of questions pop up - how would you trigger installation in a particular subsystem, would the profiles be isolated or would they share some pieces of information, etc.  
> There are many possible solutions, all with their benefits and drawbacks. The challenge is to find the one which best fits in the p2 architecture.  
> What do you think, from p2 point of view, what means "isolation"? 
> 
>> The second question that I have is more closely related to 
>> the runtime in the sense that how does the framework know 
>> what to put in each "subsystem" ? Does each subsystem need to 
>> be in its own bundles.info file, or should they actually be 
>> merged in one? Where does the description on how to configure 
>> come from?
> 
> As far as I know the decision what to put in different subsystem (or what part of the framework is seen by each of the subsystems) is responsibility of the system "owner" and could be defined via 
> configuration or programmatically. There are several open questions here: when do these visibility rules become active, could they change at runtime and what they have to do with p2 profiles (which are the ground for p2 resolving) 
> I don't think we have strict requirements how many bundles.info file we have as long as the OSGi framework is presented in a consistent way. In my opinion, it's easy and straightforward to map 
> one bundles.info to one OSGi framework and leave the hooks to deal with visibility (resolving) at runtime. In that sense subsystems and p2 will complement one another without any conflicts.
> Of course this is just one piece of the problem and should be considered in the whole context of profile isolation. 
> 
>> Finally what is interesting about this usecase is that, 
>> though the runtime aspect seems to relate to OSGi and fwk 
>> hooks, the more general usecase goes around how to describe / 
>> provision multiple applications (e.g. if I wanted to install 
>> m2e and maven in one shot, or if I wanted to install all of 
>> RAD that is composed of multiple small apps at once).
> 
> While the upper questions address the installation part, I think here we talk about publishing and associating an adequate meta information with the IUs.  Since both p2 repository and profile are used for 
> dependency resolution they need to be in sync of what a subsystem is. No idea how the magic metadata looks like though:)
> 
>> All that said, though these scenarios have been discussed in 
>> the past, I don't have a solution in mind :)
>> 
>> On 2011-01-24, at 8:38 AM, Todorova, Katya wrote:
>> 
>>> Hi, 
>>> 
>>> As subsystems are getting more and more used nowadays, here 
>> comes the logical question how to provision them in a standard way. 
>>> Eclipse Virgo using subsystems to model its runtime is one 
>> example. Currently, it is very hard to install and even 
>> harder to update a particular Virgo subsystem with existing p2 tools.
>>> Subsystem concept (rfc152) is not supposed to define 
>> initial provisioning process. Do you think p2 should be aware 
>> of subsystem existence and could fill that "gap"? 
>>> 
>>> Thanks in advance for your comments, 
>>> Katya
>>> 
>>> _______________________________________________
>>> p2-dev mailing list
>>> p2-dev@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/p2-dev
>> 
>> _______________________________________________
>> p2-dev mailing list
>> p2-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/p2-dev
>> 
> _______________________________________________
> p2-dev mailing list
> p2-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/p2-dev