[jdt-core-dev] Defining common and consistent mechanisms for JDT model a

[James Watkins-Harvey <mjameswh@xxxxxxxxx>]

Hi everyone,

There is an increasing number of projects out there that, through various approaches, intrusively alter the JDT internal structures and processes, either to support new (Java based) languages (ScalaIDE, ObjectTeams, AspectJ, etc) or add features to the processing of the Java language itself (LombokProject, Dali, some early proposal for null annotations analysis, etc).

Such extensive reuse can be understand as the consequence of the community's recognition regarding JDT's outstanding quality, stability and feature set. Eclipse users have come to expect that all code editors should offer features matching those of JDT, and yet, none have succeed at that level for any language comparable to Java, due to the immensity of the task. The XText project thankfully brought significant improvement in this regard, making it much easier to develop new simple languages with associated feature-rich editors, but the complexity of the Java language make it impractical, as well as counter-productive for the community, to develop new XText-based editors for languages that are essentially super-sets of the Java language itself. The Handly project might eventually bring up significant improvements in this direction, but too little has yet been done to make this avenue viable at present time. Developers have consequently judged pertinent to modify the JDT directly, either by maintaining a fork of the JDT's source code, or by exploiting aspect/weaving mechanisms to dynamically patch the JDT's binary code.

Unfortunately, even the most simples changes introduced in JDT are generally intrusive enough to break other components of the JDT ecosystem. The Lombok Project, for example, appears relatively conservative in how it alters the JDT, given that it does not affect in anyway the syntax of the Java language; it does not introduces any new type of nodes to the Java parsing model, and it modifies neither the binding nor the indexing processes. In most case, Lombok simply dynamically insert new fields and/or methods inside the AST tree before resolution of bindings. Yet, these alterations to the AST tree will cause errors in various refactor operations (rename field, rename method through the outline view, extract interface, etc), in the source code formatter, during search operations, etc. Developers of Lombok must therefore research for all affected processes, and introduce corresponding patches (each of them introduced through a distinct aspect interceptor) to prevent execution failures. A look at the source code of previously mentioned projects show that they all have had to fix similar issues; they also have implemented similar mechanisms and features in order to hide their alterations from various processes.

Expecting each JDT extender to develop its own fixes for each affected JDT processes (and potentially each distinct version of these processes) is highly inefficient: first, the difficulty of fixing such issues from outside (notably through an aspect) is much higher than fixing it inside the original source code. Also, extension developers generally does not have a large understanding of the JDT's source code, outside of the few packages that are directly involved in their modifications; consequently, most issues are not properly isolated before these extensions get actually used on the field by users that are totally unaware of the inner working of these extensions. Problems may even not show up until much later, for example when another plugin gets updated, and might be difficult to relate to the extension itself, for example causing issues on pure Java code. Both these extensions and JDT communities have to deal with many bug reports related to such occurrences, most often causing JDT core developers to uselessly invest time in analyzing an issue that simply can't be reproduced.

Having each extension implement its own fixes also make it highly improbable (and very risky anyway) that different extensions could successfully coexist in a same development environment. More importantly, it also compromises the whole JDT plugin ecosystem, since regular contributions (that is those that are API based and published through the plugin registry) may also be negatively affected by these extensions.

So what am I proposing?

First, rest assured, I'm not saying that JDT should strive at directly supporting such intrusive extension. Introducing into JDT new APIs that allows general language extension would, I think, be a ridiculously unwise investment; it would require considerable effort and severely compromise stability for a very long period; the resulting APIs would anyway be extremely complex, and still, would probably fail to support already existing language extensions. Aspect oriented injection into JDT’s code appears anyway to offer relatively viable means for extensions to surgically alter key JDT processes as necessary; Stephan Herrmann has offered some very neat examples in this regard, developed using ObjectTeams.

I believe however that lite modifications to the official JDT source code should be envisioned in order to reduce significantly the complexity and spread of fixes required by extension developers. An example of this would be the addition of the « synthetically generated » notion to ASTNode, to corresponding scopes, and to public tree elements. This would provide a well defined and consistant way for other participants to know about particularities of these elements, and therefore, allow [both core and external] contribution developers to themselves determine the most appropriate behaviour in regard of these particular nodes. It could also be desirable to alter various core classes in order to properly isolate into fallback methods the handling of unexpected situations that are likely to occur in the presence of extensions, for example by introducing something like "acceptUnknownNode(BaseNodeType)" in visitor interfaces, allowing for a cleaner interception point for JDT extenders.

Formalizing these synthetic alterations to JDT models would also allows the inclusion, inside core itself, of some highly desirable features common to all extensions; for example, it could be possible to generalize the "Compare with bound base" action already offered by ObjectTeams, or being able to step though the generated source code when running in debug mode... It would also provide opportunities for JDT to identify and report either a given exception or assertion failure is associated with synthetic alterations...

Given that my knowledge of JDT's internals is still rather limited at this point, it seems wise however for me to reach for the point of view of developers that are more experienced with this specific project. Am I right thinking that such commodities could be worked at core's level? Does that seem an appropriate solution to issues I have discussed about? Is there any ideas of mechanisms/modifications to be investigated in this regard? Hints? Advices? Opinions?


Thanks a lot.

James Watkins-Harvey