Bug 507795 - Write a functional replacement for project dynamic dependencies
Summary: Write a functional replacement for project dynamic dependencies
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.6   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: 4.7 M5   Edit
Assignee: Stefan Xenos CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 495062
  Show dependency tree
 
Reported: 2016-11-18 22:24 EST by Stefan Xenos CLA
Modified: 2019-01-25 17:00 EST (History)
5 users (show)

See Also:
jarthana: review+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Xenos CLA 2016-11-18 22:24:41 EST
Dynamic dependencies are a way for language tooling to inform the builder about the order in which projects need to be compiled.

However, the current approach (data which must be explicitly attached to the project via setters while holding a workspace lock) is prone to deadlocks. See bug 495062 and bug 241751.

There is also a design flaw in the current API. Since each language sets this data as a single attribute on the project, projects that support multiple languages will have this attribute overwritten by the tooling for each language.

We should replace this system with something that is purely functional. That is, the build system will invoke a function provided by the language tooling to determine the build dependencies rather than having the language tooling attach the dynamic dependencies to the project as metadata.
Comment 1 Eclipse Genie CLA 2016-11-18 22:31:30 EST
New Gerrit change created: https://git.eclipse.org/r/85332
Comment 2 Eclipse Genie CLA 2016-11-18 22:33:53 EST
New Gerrit change created: https://git.eclipse.org/r/85333
Comment 3 Stefan Xenos CLA 2016-12-19 15:29:07 EST
Jay + Sergey, could I trouble you for a code review? Since the change to resources introduces new API, I'd like the perspective of at least one other JDT committer and a reviewer who works on some non-Java tooling.

What this new API does:
- Permit callers to obtain the java class path and traverse the java model without obtaining the workspace lock. (Necessary for breaking a deadlock in the new indexer)
- Allows multiple builders to run on the same project without overwriting one another's dynamic dependencies.
- Allows dynamic dependencies to be keyed based on configuration rather than just per-project.

My main concerns:
- Since I'm reworking how dynamic dependencies work anyway, are there other use-cases I should have taken into account?
- My new API is based mainly on IProject. Should I have been using something else ? If I used something like IBuildConfiguration, would it have let people do anything that they can't do with IProject?
- How do we like the naming of the API methods/interface?
- How do we feel about the way I modified the builders extension point? Is the new configuration element in the right spot?
- How do we feel about the way I notify the IProject of changes in the dynamic dependencies (by having the laguage tooling explicitly invoke a clear* method on IProject).

Feedback on any of the above would be appreciated.
Comment 5 Sergey Prigogin CLA 2016-12-20 12:30:07 EST
Please add a N&N section about the new API.
Comment 6 Eclipse Genie CLA 2016-12-20 15:33:08 EST
New Gerrit change created: https://git.eclipse.org/r/87504
Comment 7 Dani Megert CLA 2016-12-21 04:43:21 EST
There are a bunch of tests that call and/or test #setDynamicReferences. They should be changed to use the new API or new tests have to be added for the new API.
Comment 10 Stefan Xenos CLA 2016-12-21 15:48:14 EST
> There are a bunch of tests that call and/or test #setDynamicReferences.
> They should be changed to use the new API or new tests have to be added
> for the new API.

I'll add tests for the new API. As long as the older deprecated API still exists, I'd like to keep the old tests around in their current form.
Comment 11 Dani Megert CLA 2016-12-22 09:36:40 EST
(In reply to Stefan Xenos from comment #10)
> > There are a bunch of tests that call and/or test #setDynamicReferences.
> > They should be changed to use the new API or new tests have to be added
> > for the new API.
> 
> I'll add tests for the new API. As long as the older deprecated API still
> exists, I'd like to keep the old tests around in their current form.

k.
Comment 12 Eclipse Genie CLA 2017-01-06 00:22:59 EST
New Gerrit change created: https://git.eclipse.org/r/88139
Comment 14 Stephan Herrmann CLA 2017-01-10 18:18:04 EST
Since this change some tests in Object Teams are failing.

These tests create several PDE projects and then invoke a full build. During this build lots of names are not resolvable, because the build order has not been properly computed.

From what I saw so far, computing the correct build order depends on this deleted line from ChangeClasspathOperation.classpathChanged():

  new ProjectReferenceChange(project, change.oldResolvedClasspath).updateProjectReferencesIfNecessary();
  
As a result I observe the relevant ProjectDescription having no references in #dynamicRefs nor #dynamicConfigRefs.

Unless you can explain that the test needing this line must be bogus, I hold that this change is not good.
Comment 15 Stephan Herrmann CLA 2017-01-10 18:33:45 EST
I found some more: the build order was broken only for the OT builder, not for the JavaBuilder.
Adding one line to the extension definition of the builder fixes the immediate issue:
  <dynamicReference class="org.eclipse.jdt.internal.core.DynamicProjectReferences"/>
  
I still have some regressions in the same tests, not sure if causally related.

Strictly speaking, I was working at my own risk, because JavaBuilder is internal, and reusing this for a new builder is of course not supported.

OTOH, I could imagine that other Java-like languages would use a similar approach (for me it was working very smoothly for over a decade :) ).
So an announcement of this change should definitely alert of the necessity to augment the extension definition.

I'll report back when I have analyzed the remaining regressions.
Comment 16 Stephan Herrmann CLA 2017-01-12 09:30:43 EST
I could track down the other regressions as glitches in the Object Teams tests.

=> re-closing.

Still the new extension point should be widely announced, IMHO.
Comment 17 Manoj N Palat CLA 2017-01-24 00:54:51 EST
Verified for Eclipse Oxygen 4.7 M5 with Build id: I20170123-0830