Bug 330312 - Rules for project patches in the train
Summary: Rules for project patches in the train
Status: RESOLVED WORKSFORME
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Architecture Council (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 major (vote)
Target Milestone: ---   Edit
Assignee: eclipse.org-architecture-council CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-11-16 02:59 EST by Gunnar Wagenknecht CLA
Modified: 2010-12-02 03:28 EST (History)
20 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gunnar Wagenknecht CLA 2010-11-16 02:59:49 EST
With the addition of Object Teams to the train there is now a patched version of JDT Core available in the repository. AFAIK it's the first time this happens.

We need to discuss how we should handle this. I propose something similar to the API rules for the train. 

(A) We need documentation which explicitly explain what has been patched. This
should be Bugzillas with the patches attached.

(B) We need Bugzillas opened against the patched project in order to discuss
the necessary API/extensibility changes in order to get rid of the patch.

(C) The patched project must respond to (B) within a reasonable time with a
proposed road-map towards implementing the necessary changes.

(D) The patch producers must work together with the patched project in order to
implement the necessary changes.

(E) If the changes can't be implemented in time an approval must be requested
from the AC. The AC will review and discuss the changes in detail.
Comment 1 Dani Megert CLA 2010-11-16 03:05:08 EST
In my opinion only the owning project must be allowed to patch its code. Exceptions must be granted by the affected project's PMC.
Comment 2 Gunnar Wagenknecht CLA 2010-11-16 03:13:42 EST
A few comments to the steps outlined in comment 0. 

I think that the process should be a positive process. As such, we should not prohibit patched projects from the train. But we should take any reasonable action to get rid of the patches by implementing them.

Patches are a good. They are healthy for the Eclipse ecosystem. They express interest in a project and they demonstrate contribution activity. The usually describe a flaw (eg., bug or missing functionality) in the patched project. 

As a community we should encourage acceptance of patches. A project needs contributors. Contributors must get a confirmation that their contributions are welcome.

A patch is not a fork. It should be reasonable in terms of size and constrained to a specific functionality that fits to the project scope. It must not change the goals of a project completely.
Comment 3 Dani Megert CLA 2010-11-16 03:16:56 EST
Please don't mix things here. I am absolutely in favor or getting patches but they must be provided trough the project which reviews the patch and then commits it to the project. Otherwise the repository ends up being a zoo and all testing efforts that were made by the project are doomed.
Comment 4 Dani Megert CLA 2010-11-16 03:21:07 EST
BTW: there's also a legal aspect here: each project keeps an IP log with the contributors and its contributions. Allowing other projects to patch a project in the official repository also dooms that process or at least makes the process much more complicated.
Comment 5 Gunnar Wagenknecht CLA 2010-11-16 03:35:26 EST
(In reply to comment #1)
> In my opinion only the owning project must be allowed to patch its code.

I think that's impossible to implement. You just can't control what people are doing with your code. For example, every Linux distributor patches a kernel or some other product they ship in their distribution. None of these patch are "under control" by the owning project developers.

> Exceptions must be granted by the affected project's PMC.

I'd like to see the AC take a more active role here. A PMC can raise the voice and vote in the AC. The view of PMC is targeted towards the project within its scope. As with the example Object Teams / JDT we already have a case that extends a single PMC's scope. That's why I think the AC is a much better fit.

According to the voting rules, the PMC may still do a veto vote. However, given the nature of such a strong vote, the issue should be discussed extensively and in public first.

(In reply to comment #3)
> Otherwise the repository ends up being a zoo and all
> testing efforts that were made by the project are doomed.

A project tests and a releases a specific version. That is documented in the release notes. It's up to the patched project how they handle patched versions produced by others. Typically, they don't support them at all.

(In reply to comment #4)
> BTW: there's also a legal aspect here: each project keeps an IP log with the
> contributors and its contributions.

The patched project isn't affected. According to the IP process it's the responsibility of the patching project to document all dependencies. This includes EPL dependencies to other Eclipse projects. Of course, the IP of the produced patch is within the producing project's scope. Anyway, please open an IPZilla question to further analyze/discuss/clarify the legal aspect.
Comment 6 Dani Megert CLA 2010-11-16 03:45:25 EST
The only thing what I want to make sure is that when someone builds JDT (or any other project) from the official repository, he gets JDT and not some unexpected piece of code. If another project or distributor offers a patched version at their download page/site then that's a different story.
Comment 7 Stephan Herrmann CLA 2010-11-16 04:49:56 EST
I was expecting this discussion much earlier, actually.
We (Object Teams) have always openly expressed what techniques
we use in our features and shipping a patch against JDT/Core was
always on the list.

I'll be back with a more detailed answer to the concerns expressed 
in this bug within the next 1 or 2 hours.
Comment 8 Stephan Herrmann CLA 2010-11-16 06:28:32 EST
Let me start by saying that I appreciate this discussion to happen.
After reading the comments and after actively working in this field
for several years now, let me try to structure the potential area
of discussion into a few (independent?) dimensions:

Patch vs. Branch:
Technically, we are speaking of a patch feature (a feature having
the capability to replace a plugin), but the semantic cut-off line
between patch and branch is certainly difficult to draw. In the case
of the Object Teams Development Tooling we clearly have a branch of
the JDT/Core (as described in all relevant documents starting from
the original project proposal[1]).

Eclipse Project vs. Release Train:
To what degree are the concerns expressed here tied to participating
in the simultaneous release train vs. general rules for all Eclipse
projects? The original bug 330288 has been identified as a bug in the
b3 aggregator. Thus I see no *technical* reason for defining stricter
rules regarding the release train. The question remains what statements
does Eclipse make by (a) accepting a project that distributes a patch
feature and by (b) including such project in the release train?

Legal Issues:
All code of the Object Teams Development Tooling has passed IP analysis.
Does this leave any legal questions unanswered?

Avoiding bad Reputation:
Clearly the JDT has good reasons to avert any negative effect on its
reputation. It must be avoided that arbitrary patches go unnoticed in
spoiling the quality of their code, thus causing harm to the JDT project.
Here we should distinguish:
- How to avoid wrong blame: a problem potentially introduced by a patch
  must never appear as being caused by the original plugin.
- How to avoid quality degradation in the first place.

I think these are the top categories to discuss.
Am I missing any points?

Details to follow in subsequent comments.

For any issues relating to the user's view of all this 
please feel free to join in on bug 316702.

[1] http://www.eclipse.org/proposals/object-teams/
    See 3rd paragraph under "Relationship with other Eclipse Projects"
Comment 9 Krzysztof Daniel CLA 2010-11-16 06:59:25 EST
Let me add my 2 cents from support perspective:

Patches are generally used to provide bug fixes on the top of the product. OSGi/p2/PDE restrict that only one feature patch may be available for a feature.
Supporting product through features patches will be impossible if they will be present in an Eclipse release.

More, discovering who built the plugin, and tracing the root cause of a problem, will also be nightmare. Even if the plugin was built at Eclipse org, and properly described, try to explain to the customer that particular version of JDT is not under the control of JDT (for example). Also, if more than one project wants to overwrite JDT, who will do that?

Not to mention that the idea of patching features in Eclipse Train Release looks like a problem between teams and generally should be solved by the technical discussion (what can be implemented or cannot) and not by the overwriting policy.
Comment 10 Gunnar Wagenknecht CLA 2010-11-16 07:08:13 EST
Stephan, this request is *NOT* about your specific case. This request is about a general process for patched features. I'd like to keep the issue at hand out of the general discussion for now.

(In reply to comment #6)
> The only thing what I want to make sure is that when someone builds JDT (or any
> other project) from the official repository, he gets JDT and not some
> unexpected piece of code. 

That's a reasonable request. I think we may need some enhancements in p2 and/or other areas along the road to better visualize/address this. I opend bug 330338 to address this.
Comment 11 Stephan Herrmann CLA 2010-11-16 07:19:57 EST
(In reply to comment #10)
> Stephan, this request is *NOT* about your specific case. This request is about
> a general process for patched features. I'd like to keep the issue at hand out
> of the general discussion for now.

Well, both of us are using Object Teams as the example, right? :)

I'm interested in both: the general discussion and a solution regarding
Object Teams. Should I open a new bug for discussing issues specific to
Object Teams or are you fine with our example as an illustration of
potential issues?
Comment 12 Stephan Herrmann CLA 2010-11-16 07:30:08 EST
Patch vs. Branch:

The process in comment 0 looks great. I'd be happy to participate in
such process. I see two reasons why it isn't working already:

- Bug 36939 is the classic example of a needed agreement just not 
  happening in more than seven year's time. I'm not sure what exactly
  caused failure on that issue. Maybe we could use that bug as a
  witness for debugging the process.

- Some changes are just beyond the scope of the original project.
  Merging the Object Teams-related patches back into the JDT/Core
  would require roughly 2000 bugzillas to be opened, processed and
  resolved. This would stall JDT and Object Teams for several months.
  So, the proposed process cannot possibly work for actual forks.
  
One word regarding the necessity of forks: the better part of a full
generation of researchers on object oriented programming languages has
developed their programming languages as extensions of Java. They did/do
so in order to make all the coolness in libraries and tooling available
to users of the new language, too. Without doing so, any comparative 
evaluation of new programming languages would be futile (not to mention
the 'likelyhood' of a language with none of that coolness being adopted 
by industry).
At the compiler front efforts have been made to create extensible tools,
where new language features can easily be added. JastAdd [1] is an example
in this group, but no such tool exists in Eclipse land. Approaches like
IMP and Xtext extend this vision towards typical IDE functionality.
However, none of these projects are even close to supporting the 
development of a JDT-quality IDE for a Java-based programming language.
I do hope that such projects will eventually get there but for the years
to come we cannot count on the solution.
Ergo, we have no platform with built-in extensibility for building IDEs
for Java-based (or Java-like) programming languages. 
Adding such extensibility to the JDT would be extremely cool and
extremely difficult.

Interestingly, a few projects have succeeded to accomplish the 
unanticipated: leverage the JDT for Java-based/like languages.
Different projects use very different means to accomplish, but all
have in common that some bits in a running instance of the JDT are
not those bits that the JDT team has shipped.

The point in this comment is: if Eclipse would over-cautiously protect
the bits of the JDT, it would loose any connection to all that research
described above.

[1] http://jastadd.org/
Comment 13 Gunnar Wagenknecht CLA 2010-11-16 07:57:43 EST
(In reply to comment #12)
> - Bug 36939 is the classic example of a needed agreement just not 
>   happening in more than seven year's time. I'm not sure what exactly
>   caused failure on that issue. Maybe we could use that bug as a
>   witness for debugging the process.

That's a perfect example on why we need a process. Obviously the project came up with a road-map (C). Also, part of (D) happened (although not with somebody from  ObjectTeams). However, the patched project failed to work with contributors together to incorporate the patch. That's why the process explicitly includes (E). We need to have the ability for projects to included patched versions in their deliverables (even if part of the train). Thus, the project could now go to the AC and request approval for submitting a patched version with the patch from attachment XYZ applied.

>   So, the proposed process cannot possibly work for actual forks.

Correct. It's a process for patches (such as bug 36939). If, however, you created a fork this process won't help. But a fork typically comes with different identifiers (eg., different bundle symbolic names, different bundle vendor, etc.).
Comment 14 John Arthorne CLA 2010-11-16 09:22:10 EST
My starting position is that each project owns its bundle/package namespace, and no other Eclipse project should be allowed to release bundles/packages under that namespace. Ever. For example, only the Web Tools project should be allowed to release bundles/packages under the org.eclipse.wtp namespace. The support/provenance problem in comment #9 is real - if eclipse.org ships multiple bundles of the same name, produced by different projects, containing different bugs/features, then figuring out how to report problems becomes far more difficult. 

This becomes especially problematic if the divergent bundles are in the same repository and have the same three part version number. Most of the dependency resolution technology in our install and runtime technology could pick either one of those copies, depending on which one has a higher qualifier on any given day. The consumer should be able to choose whether they want the original bundle or the forked bundle, but this is impossible if the bundles have the same name and version. There is no version range that the consumer can specify that would include versions of the original bundle but exclude versions of the forked copy.

Now, *forking* code from one Eclipse project to another is perfectly acceptable. In fact there are many forks of JDT in the release train today - even CDT started out as a JDT fork. However, forks are done by renaming bundles/packages, so that consumers can opt into the fork or select the original at their own discretion.
Comment 15 Stephan Herrmann CLA 2010-11-16 09:26:46 EST
(In reply to comment #13)
> (In reply to comment #12)
> > - Bug 36939 is the classic example of a needed agreement just not 
> >   happening in more than seven year's time. I'm not sure what exactly
> >   caused failure on that issue. Maybe we could use that bug as a
> >   witness for debugging the process.
> 
> That's a perfect example on why we need a process. Obviously the project came
> up with a road-map (C). Also, part of (D) happened (although not with 
> somebody from  ObjectTeams). 

Indeed this whole example doesn't relate to Object Teams :)
Still an interesting example, right?

> However, the patched project failed to work with
> contributors together to incorporate the patch. That's why the process
> explicitly includes (E). We need to have the ability for projects to included
> patched versions in their deliverables (even if part of the train). Thus, the
> project could now go to the AC and request approval for submitting a patched
> version with the patch from attachment XYZ applied.

All this makes sense.

If we agree that (A)-(D) are clearly preferable over (E), (E) can be used
as a transitory fall-back if the others fail.

From there on, focus should be on actually enacting (C), which is where
many bugs to-date fail (incl. bug 36939 - when considering the "within a
reasonable time" part). 
Any ideas?
Comment 16 Martin Oberhuber CLA 2010-11-16 09:29:19 EST
+1 I fully second John's comments, adding a different angle:

It may not always be possible to avoid patches/forks since the original project cannot satisfy all the needs of all possible clients. It's good to feed back and discuss requirements ("projects must work together" is a release train requirement) but there is no guarantee that all requirements can be resolved timely.

That being said, the important point is that clients know what they get. As Dani said, if I select JDT I want to get JDT (ie an original, untainted JDT). That's partially the issue discussed in bug 330288.

If some clients require a modified JDT for whatever reason, that must be clearly marked as a modified JDT. I don't feel that patches are a good way of marking such modifications, because (as per comment 9) only one feature patch is possible. This may lead to unexpected conflicts downstream.
Comment 17 Gunnar Wagenknecht CLA 2010-11-16 09:43:09 EST
(In reply to comment #16)
> That being said, the important point is that clients know what they get.

Agreed. 

I was thinking of other ways (eg. vendor info, project specific signing) but John's statement is the better (and clearest) solution, i.e. the namespace of a project is owned by that single namespace.

This raises the effort for a single patch. But may be this is reasonable if (A) - (D) fail. IMHO, having a process is still a good thing as it would encourage people to actually work together on the issue at hand.
Comment 18 Gunnar Wagenknecht CLA 2010-11-16 09:44:08 EST
> [...] i.e. the namespace of a
> project is owned by that single namespace.

"by that single project"
Comment 19 Stephan Herrmann CLA 2010-11-16 09:53:26 EST
(In reply to comment #14)
> This becomes especially problematic if the divergent bundles are in the same
> repository and have the same three part version number. Most of the dependency
> resolution technology in our install and runtime technology could pick either
> one of those copies, depending on which one has a higher qualifier on any given
> day. The consumer should be able to choose whether they want the original
> bundle or the forked bundle, but this is impossible if the bundles have the
> same name and version. There is no version range that the consumer can specify
> that would include versions of the original bundle but exclude versions of the
> forked copy.

I don't see the technical issues you mention, because users are free to 
chose the features to install. Only when (directly or indirectly) selecting 
a patch feature the forked bundle will be pulled in. Otherwise the exact
dependency from the original feature to the original plugin will prevent the
forked plugin from being pulled in.
While we are lacking experience with both variants being hosted in the same
repository, the general scheme has been working just fine for years.

Perhaps you are talking about a scenario where the original plugin is not
part of any installed feature but pulled in implicitly via Require-Bundle
or Import-Package? That scenario would have to be reconsidered, indeed.
 
> Now, *forking* code from one Eclipse project to another is perfectly
> acceptable. In fact there are many forks of JDT in the release train today -
> even CDT started out as a JDT fork. However, forks are done by renaming
> bundles/packages, so that consumers can opt into the fork or select the
> original at their own discretion.

Alright, given this definition of forking we have three flavors:
(A) patches as subject to the process proposed by Gunnar
(B) forks which have been decoupled by moving to a new namespace, not an issue
(C) forks within the original namespace.

Object Teams has good reasons for living in (C). On the one hand I know 
we're not the only project in this space (although the other one I'm 
thinking of is not hosted at Eclipse.org). On the other hand I get the 
feeling that Object Teams is entering territory where it's too early to
define general rules that everybody can agree on.
For that reason I've precautionarily started the process of requesting an
exception from the process discussed in this bug. This should free the
current discussions from too many specific details.

Let's hope the new process will help some projects if not Object Teams.
Comment 20 Gunnar Wagenknecht CLA 2010-11-16 10:26:29 EST
(In reply to comment #19)
> Alright, given this definition of forking we have three flavors:
> (A) patches as subject to the process proposed by Gunnar

I don't like to see the process as a flavor of forking. What I'd really like to see is people working together to get the patched project extensible in a way that obsoletes the patch.
Comment 21 Stephan Herrmann CLA 2010-11-16 10:36:33 EST
(In reply to comment #20)
> (In reply to comment #19)
> > Alright, given this definition of forking we have three flavors:
> > (A) patches as subject to the process proposed by Gunnar
> 
> I don't like to see the process as a flavor of forking. What I'd really like to
> see is people working together to get the patched project extensible in a way
> that obsoletes the patch.

Sorry if this was ambiguous, I didn't mean to say that (A) is forking,
only (B) and (C) are.
Comment 22 Andrew Clement CLA 2010-11-16 11:12:49 EST
Thought I would comment on this as I am involved in two other projects that
work with JDT in this kind of way (Stephan eluded to one of them) - one is at
eclipse.org, one is not.

AspectJ - a tools project at eclipse.org.  This currently does what we are
calling the 'fork' approach here, taking a cut of org.eclipse.jdt.core and
changing the namespace to org.aspectj.org.eclipse.jdt.  The changes made by
AspectJ to jdt.core are quite radical and I can't immediately see an easy way
they could be rolled into jdt.core over time to remove the need for a forking
(or patching).  Some of them could be, but like many on this thread I don't
want to pollute jdt.core and affect the original project mission/goals or force
jdt developers to debug through AOP related stuff.  A forked approach has
proved really nightmarish in places - when you have something very similar to
java that isnt, you end up mapping/converting a lot back and forth between
org.eclipse.jdt.* and org.aspectj.org.eclipse.jdt.* - it's horrible.

Groovy-Eclipse - hosted at codehaus.  This patches jdt.core (feature patch). 
Learning from AspectJ, when I implemented this patch I've done the minimum
required to open it up: (1) avoid groovy specific changes in the patch (2)
patch to allow extensibility outside of jdt.core (e.g. make methods protected
rather than private, extract certain bits of inline function into standalone
methods, so they can be overridden).

In that groovy-eclipse case, I'd say 80-90% of groovy-eclipse changes could be
dropped into jdt.core and not affect/pollute it.  The remaining 10-20% would be
tricky but possibly doable.  I'd love to take this approach and get rid of the
need to patch.  We haven't been actively doing this because of issues like bug
36939 - if it takes multiple years to get it in, there isn't really the impetus
for me to do the work of creating patches/tests/etc.  Following (A)->(D) (as proposed in the first comment) would be great from my point of view.
Comment 23 Andrew Clement CLA 2010-11-16 11:33:50 EST
I should add that AspectJ/AJDT are not on the train right now (and of course groovy eclipse is not).  AJDT *was* on the train during the time in which bug 36939 has existed - it didn't seem to help progress the issue that we were on it.  Of course I appreciate the bug summary is rules 'for the train', my intent was really to talk about experiences with trying to fork and trying to patch and sharing my desire to get reasonable changes into JDT in a timely way.
Comment 24 Stephan Herrmann CLA 2010-11-16 14:06:37 EST
Hi Andy,

thanks for sharing your experience.

What you write about the AspectJ approach confirms me in not wanting
to go that road for Object Teams.

The Groovy-Eclipse story sounds much nicer indeed. It's on my agenda
for discussion with the others from JDT/Core how we could help you
to avoid this patch altogether. Just as I'm still new to that team I don't 
want to promise too much too quickly.

Let's keep in touch on this.
Comment 25 David Williams CLA 2010-11-17 02:10:14 EST

I am not sure I can add much constructive to this conversation, that has not already been said in previous comments, but ... that's never stopped me before. :)

I agree with those that have said a package/bundle namespace belongs to the project it has been assigned, and other projects can not use it. In any way. for any purpose.

I suspect this is confusing because there is a sort of romantic myth that "... its open source, we can do whatever we want ...". But that's not true, as others have pointed out. The EPL does allow others to modify source and redistribute, but that is a legal statement about the license governing its use, between the Eclipse Foundation, and those that use/consume it. It doesn't say it is ever a good or desirable idea to do that (it is not) ... and it does not say if it is ever permissible to do so at the Foundation itself. It is not; not in a release train; not even in a release, in my opinion.

I think it has never been prohibited by an explicit rule, simply because its obvious, to most, that there is "ownership" of an assigned package/bundle name and others can not reuse it, in my opinion.

The only way this seems technically feasible, to some, is by a bit of mis-use (abuse?) of version qualifiers. There is a type of a "package qualifier" snuck in the version qualifier in this case. So that is conceptually wrong from the start. I think practically that sort of (mis) use of versioning can never be made to work in a large complicated community and eco- system such as Eclipse, in my opinion.

One more point on the "support issue". It is multiplicative. Not only does it make it harder (or impossible) for JDT (in this example) to work on reported bugs, and to provide patches, and maintenance, but others, such as WTP, rely heavily on JDT so they too would have complications trying to provide support. It is bad enough already when our committers spend hours investigating a bug report, only to discover the error is occurring in some "custom" modifications and have to close the bug as "not eclipse". Let's not make it worse.

I know everyone means well, but I think great care is needed in adding processes as stated in comment #0. Statements such as "If the changes can't be implemented in time an approval must be requested from the AC." sound rather ominous -- almost corporate :)  I think committers to a project are in complete control of their own project and not sure there are grounds for others in Eclipse to say what has to be implemented when, in my opinion.

Of course, the Architecture Council members can clearly volunteer to help review patches, improve them, as well as cajole and remind committers of the importance of accepting patches and considering requirements from the community. And, I am sure, the architecture council members have the skills and motivation to do that in a way to be perceived as helpful, instead of bossy.

So where to next? I think if the namespace issue is settled, then this bug could be transformed into "How can AC help projects with their patches and requirements"?

Plus, I'm sure, there are some on this discussion that'd be glad to help/mentor the object team project an avoiding the need to use the same namespace. 

Sorry I don't have any concrete solutions ... it is a hard problem. 
If I can help in any way, let me know.
Comment 26 Gunnar Wagenknecht CLA 2010-11-17 03:28:12 EST
(In reply to comment #25)
> I think it has never been prohibited by an explicit rule, simply because its
> obvious, to most, that there is "ownership" of an assigned package/bundle name
> and others can not reuse it, in my opinion.

Maybe it wasn't that obvious. Otherwise we wouldn't have an issue as well as a discussion.

> So where to next? I think if the namespace issue is settled, then this bug
> could be transformed into "How can AC help projects with their patches and
> requirements"?

Looks like we reached an agreement on the namespace issues. Do we want to state that explicitly somewhere? I prefer we do. Or is this captured under the "play well with others" umbrella?
Comment 27 Jeff McAffer CLA 2010-11-17 04:11:54 EST
I'm sympathetic to the patch management issue but do think we are going a little too far with some of the other arguments.  Basically you can modify any bundle's code using aspects.  Should we say that people cannot supply aspects that weave on org.eclipse packages?  Would be a little odd given that we (Eclipse) host AspectJ, AJDT and Equinox Weaving. (Good for the community but not for us?)

It is true that allowing modification (by any means) may complicate maintenance.  So do aspects/weaving.  So does exporting "internal" things for that matter. JDT could start failing just as easily and just as inexplicably by someone calling internal code and messing with internal structures.   We have tools that tell people that they shouldn't do that (reference internals) but don't exclude people who do from the releases.  We do these things to enable innovation.

The particular patch mechanism that OT uses is unfortunate since it has the namespace issue and inhibits other patches being installed. If JDT puts out a new patch and that can be installed in favor of the OT patch that seems ok.  OT might be broken but they can supply a new patch as Stephan points out.

IMHO the key is that users should not get these mods for free or by accident.
Comment 28 Krzysztof Daniel CLA 2010-11-17 04:36:21 EST
(In reply to comment #27)
> IMHO the key is that users should not get these mods for free or by accident.

Green light for feature patches in a release train leads us nowhere. If the amount of work is equivalent to 2000 bugzillas, either additional commiter to JDT should be founded by an interested company or a plugin should be forked within non-interfering namespace. 

Then we should let people vote with their dependencies ;-).


Another issue is that some projects may be too closed for contributions. Maybe we could make this transparent, for example publish statistics like number_of_accepted_contributions/number_of_total_contributions for each project in a release train?
Comment 29 Stephan Herrmann CLA 2010-11-17 07:58:09 EST
(In reply to comment #27)
Let me add just one more "bad" mechanism to the list:
  setAccessible(true)
This is the stealth mode of breaking the rules,
because the fact of (mis)using reflection is burried deep 
within the code and no-one can tell from the outside.
Comment 30 Chris Aniszczyk CLA 2010-11-17 14:43:18 EST
(In reply to comment #27)
> IMHO the key is that users should not get these mods for free or by accident.

I agree. When we go to site categorization, can't we just hide the OT JDT contribution so people wouldn't accidentally get it unless explicitly installing OT?

(In reply to comment #20)
> I don't like to see the process as a flavor of forking. What I'd really like to
> see is people working together to get the patched project extensible in a way
> that obsoletes the patch.

In reality, some of us who have been in the Eclipse community for awhile should know this is harder for certain pieces of Eclipse (e.g. JDT)...

https://bugs.eclipse.org/bugs/show_bug.cgi?id=36939 (7 years and counting...)
Comment 31 Gunnar Wagenknecht CLA 2010-11-17 15:05:54 EST
(In reply to comment #30)
> (In reply to comment #20)
> > I don't like to see the process as a flavor of forking. What I'd really like to
> > see is people working together to get the patched project extensible in a way
> > that obsoletes the patch.
> 
> In reality, some of us who have been in the Eclipse community for awhile should
> know this is harder for certain pieces of Eclipse (e.g. JDT)...

Indeed. "Play well with others" is bi-directional. We need to build some positive pressure on projects and teams. Eclipse is all about community. However, building a community requires both sides to approach each other. At some point, a "resource issue" sounds just ridiculous if there are 3 people on a bug consistently updating patches.

Jeff made a good point about innovation. That's why I still like (E). The namespace issue is real. But prohibiting patch features is the wrong signal.
Comment 32 John Arthorne CLA 2010-11-17 16:27:40 EST
We discussed this topic in today's Eclipse Project PMC call (http://wiki.eclipse.org/Eclipse/PMC).

The short summary is that we cannot live with alternate copies of bundles in our  namespace being present in the release train repository. This is one of the primary delivery mechanisms for our code, and the risk of our consumers not getting what they want/expect is too great. We cannot pollute our delivery channel in this way, and we will be forced to withdraw from the release train if we can't ensure this for our consumers.

Having said that, we're open to just about any other possibility for project such as OT/J to deliver their functionality: forking in a different namespace, patches hosted elsewhere, any form of install-time/load-time/runtime weaving, working towards putting hooks in JDT that prevent the need to fork, or any other suggestion people can come up with. The important point is that an installer or OSGi resolver attempting to get "org.eclipse.jdt.core" from the set of bundles in the release train can only get one answer.

We really are willing to discuss and work on a technical solution that allows OT/J to release in a way that doesn't add risk for our consumers. For example last year we discovered a project shipping a copy of JDT in the release train and we ended up adding a new artifact in our build to suit their requirements (bug 315844). In this case perhaps forked/patched projects can be contributed to the repository as binary artifacts with different names, but are then installed as bundles under the forkee's namespace. Should we open a separate bug to continue that discussion, or can this bug be diverted to that topic?
Comment 33 Gunnar Wagenknecht CLA 2010-11-17 16:31:00 EST
(In reply to comment #32)
> [...] Should we open a separate
> bug to continue that discussion, or can this bug be diverted to that topic?

Please open a separate issue for discussing possible solutions for OT/J.

@AC members, should we put this issue on next calls agenda?
Comment 34 Stephan Herrmann CLA 2010-11-17 18:36:09 EST
Given that load-time weaving seems to have the blessing here's an
additional heads-up:

Load-time weavers typically use the Equinox Adaptor Hooks,
but due to bug 329784 any hook that is registered using the corresponding
p2 touchpoint action will actually overwrite any existing hooks.

This may also affect other list-type properties. 

IMHO p2 could give some more help towards "play well with others".
Comment 35 Olivier Thomann CLA 2010-11-17 20:58:17 EST
(In reply to comment #33)
> Please open a separate issue for discussing possible solutions for OT/J.
> @AC members, should we put this issue on next calls agenda?
I am definitely in favor of discussing what could be done to solve the issue for OT/J. I opened bug 330534 to track this down.

For me as long as the user knows exactly what ends up being installed and he/she knows that he/she gets a patched version of JDT/Core when OT/J is installed, this is fine.

For me, the release train is about getting multiple projects that are based on the same roots from the same repository. It is not about integration of different projects.
Comment 36 Martin Oberhuber CLA 2010-11-18 11:05:32 EST
This bug has originally been about "rules for project patches in the train".

Reading through the comments, it looks like there is abounding consensus from all the AC and PMC members commenting on this bug, that only the project originally producing a bundle may own that namespace and thus has the sole right delivering patches for that bundle on the train (I'm not going to repeat all the arguments here).

We have bug 330534 as a follow-up to help OT find a solution that helps them remain on the Indigo train (assuming and hoping that OT still wants to stay on the train ... Stephan, a comment of yours on bug 330534 would help). AC members, please CC on this bug to help finding this solution.

This leaves one task to be done for this bug before it can be closed -- documenting the resolution. IMO it could be a sub-section of the "Do No Harm" section on the Indigo Must-Do's. Could somebody draft something or could we claim it's implicit from the existing text? See 
http://eclipse.org/indigo/planning/EclipseSimultaneousRelease.php

I would also appreciate an explicit Indigo Rule along the lines of Gunnar's original items (C) and (D), forcing a patched project to respond and the patcher to collaborate with the patched... since that would constantly remind, if not force us to avoid forking where possible at least withing the Eclipse.org project space; and find smart solutions that make Eclipse as a whole more extensible and thus interesting for innovation. Yes, bug 36939 is a very sad example where this has failed, but I do think that the Eclipse Platform team has learned a lot and is now much more open than it was 7 years ago - like also John's comment 32 publicly shows.

From an Architecture Council point of view, I can only encourage people to bring up / escalate issues of non-progress related to cross-project architectural themes with the AC, see
http://wiki.eclipse.org/Architecture_Council#Getting_in_Touch .
Comment 37 John Arthorne CLA 2010-11-19 09:54:28 EST
Stephan, I wrote this reply to your recent blog post. Your blog seemed to eat the comment, but I thought it was worth repeating it here:

>From two days distance I can already laugh at one implication of the central >argument, paraphrased as: the JDT/Core team must be protected against harmful >actions from the OT team.

It's too bad you got this impression, because I don't think this was anyone's position. The central point is that a consumer that wants to obtain software X should not obtain software X'. This is not a qualitative statement about X versus X', but about consumers being able to specify/control their dependencies. In the reverse direction, if a consumer really needs the OT/J version of jdt.core, I suspect they would be quite broken if they got the Eclipse Platform's version of jdt.core. Both sets of consumers need to be able to specify those dependencies.
Comment 38 Stephan Herrmann CLA 2010-11-19 14:25:08 EST
(In reply to comment #37)
John, my apologies, I was clearly overreacting in the paragraph you quote.

Even if any comment had that connotation, that never was the main line
of argumentation, agreed.

Your description is perfectly in line with the goals in bug 330534.
Comment 39 David Williams CLA 2010-11-30 03:39:11 EST
(In reply to comment #36)
> This bug has originally been about "rules for project patches in the train".
> 

I've put this on the Planning Council's agenda for 12/1 meeting. 

I'd be willing to add a small clarification on "only one namespace" in repository/release in "do no harm" section if others on council agree. I'm less sure about specifying rules for how projects are to interact with each other, to quote, along the lines of ... "Gunnar's original items (C) and (D), forcing a patched project to respond and the patcher to collaborate with the patched".  I'm not sure we have other rules for how projects are to interact with each other, but I'll check with other PC members and glad to hear their input. I suspect such "right behavior" is the province of mentors, Eclipse leaders, and the Architecture Council itself, rather than rules of the yearly Simultaneous Release. 

Also, I have opened a "process" bug, since I think it might help codify our assumptions and rules to say more about project namespaces in the EDP. 
See bug 331397 and comment there if you think there is a place to say something in EDP. 

Thanks,
Comment 40 David Williams CLA 2010-12-02 01:14:31 EST
(In reply to comment #39)
> (In reply to comment #36)
> > This bug has originally been about "rules for project patches in the train".
> > 
> 
> I've put this on the Planning Council's agenda for 12/1 meeting. 
> 

To summarize, no action or changes are planned based on Planning Council's discussion. The main points of the discussion were a) current known case that gave rise to this is exceptionally rare, and we probably don't need rules about very rare events, b) the main principle is "not to interfere" and is already covered by the general "play well with others" descriptions, c) it is well known (described by one member as 'common sense') that one project can not casually use another's namespace at Eclipse.org and was not anticipated to be a problem except in exceptionally rare cases, d) this was (mostly) a project-to-project issue and it is up to ObjectTeam and JDT to work out and come to some joint resolution they are both happy with (and, if that's not possible, the normal escalation processes should suffice). 

Everyone agreed current case is important to resolve, and is complicated, but that there was not sufficient reason for any specific "thou shalt" or "thou shalt not" rules about it. 

And that's not to meant to day all these discussions haven't been good and useful, and to some extent, the discussions here in bugzilla are probably sufficient to guide behavior in future cases. At least that's what we hope. 

Thanks everyone for your comments.
Comment 41 Gunnar Wagenknecht CLA 2010-12-02 03:28:41 EST
Thanks David!