Community
Participate
Working Groups
This is an enhancement request for the next version of JDT after 3.2. The idea is to produce a pattern language, rules engine, and user interface to allow the Java developer to perform queries on and make safe modifications of their code code. It's a superset of the current refactoring because ideally current refactorings could be expressed this new system, forming a nice set of initial examples. The users could then adapt those rules to their own purposes or come up with completely new ones. It's also (at least partially) a superset of some of the more elaborate JDT compiler diagnostics that aim to find bugs through static analysis of the code. One use case for this technology besides refactoring would be sophisticated i18n checks. SAS has extended the Eclipse Java compiler internally (see bug 38332) to diagnose a whole bunch of patterns in Java source (and also in HTML and other files) that could potentially cause localization problems. (I'd need to seek permission but it might be possible for us to contribute these i18n rules to Eclipse if an expressive enough rules language were implemented.) Another use case would be to use it as a platform for tools like PMD and Instantiations CodePro auditing. If everybody is speaking the same (rule/pattern) language, so to speak, there will be a great opportunity for innovating, combining and sharing coding patterns and antipatterns within one framework instead of everybody having to do the same hard work of language parsing, rules engines, batch substitutions, hooks to tests, and so forth over and over. The idea is based on research performed by Sun Labs starting around the year 2000 under the name Project Jackpot. Project Jackpot was demonstrated at JavaOne 2006 by Tom Ball running as a NetBeans plug-in. See the references below for more information. In Jackpot, they are using public APIs on the javac compiler to parse and reason about the user program. An Eclipse equivalent would naturally use the Eclipse compiler. Although this is not a requirement, it would be nice if some code or ideas could be shared between the two projects, such as the syntax of the rules language (which would let rules be posted on bulletin boards and shared between the two IDEs). In addition, again this isn't a requirement but it would be nice for the system to be extensible to other languages as long as that doesn't make it tremendously complicated. This would be something that Jackpot doesn't do. I'm thinking especially of C/C++, but also JavaScript, JSP, Groovy, and maybe even JRuby. Note: According to the FAQ, Jackpot is not open source yet but plans to be: "The biggest barrier to Jackpot being open-sourced is being removed with the inclusion of JSR 199, JSR 269 and the new Tree API in Java 6 (Mustang). These APIs provide a public read-only model of Java source code, and are supposed to release the end of this year [2006]." There might also be some synergy with the Eclipse Monkey project, for example running rules under script control, or even writing rules in a script language. However I'd file that under 'nice to have' for now. References: Jackpot home page: http://jackpot.netbeans.org/ (has binaries and sample rules files) Tom Ball's blog: http://weblogs.java.net/blog/tball/ Use cases, scenarios, and user interface ideas: http://jackpot.netbeans.org/docs/jackpot-uispec.html
Thanke Ed for the bug and all the pointers. We're looking at 'Generic search' and 'Generic Refactorings' for quite a while. We had it on the 3.1 and 3.2 plan, but never had the time or the full vision for it. Biggest problem is the user interface. Finding a simple but powerful way of defining queries is the challenge. What we're most intersted are, are concrete examples of queries that you want to make. So please list them here so we see how simple or how complex we have to be.
I think TPTP's Static Analysis could also benefit from this.
Does this functionality has already been added to Eclipse? I spent more than one year working on a project of a AspectJ Transformation System, that allows the users to define it's own refactorings in a language that is very similar to AspectJ. Of course, it works to Java as well. The page of the project is http://www.cin.ufpe.br/~jats/ajats/. I'm not really sure if this is what you were looking for, but if it is, I'll be really glad to implement it and make the adaptions needed as a Google Summer of Code project.
There's nothing new from our side on this idea and also no plans (we don't have the resources at this time) What you're doing looks interesting, similar approach like Jackpot? (see link in comment 0). Looks like a good idea to apply to SOC. Have a look at http://www.eclipse.org/projects/project_summary.php?projectid=technology.soc
I had a look at Jackpot project and is quite different than what i'm doing. As far as I could see, Jackpot specifies rules of transformations. For example, ($T)$a => $a :: $a instanceof $T; is used to removed unnecessary casts. AJaTS works quite different. With AJaTS, you specify a source file and a destination file. The source file matches with the AspectJ/Java program that you want to transform and the destination file is the skeleton of the program that will be the produced by the transformation. I think maybe AJaTS can be used both in the Refactor option and the Source option in Eclipse. With AJaTS, you can easily define the source transformations that are described there as generate get and set, generate construtor using fields. With AJaTS, we could allow the users to define it's own transformations. We can have a website where the users can add the transformations they define and other users can browse to see what transformations are available if they don't want to create their own. I don't know if the Eclipse project is interested in something like that. What can you tell me about it?
Hi Patricia > transformations are available if they don't want to create their own. What > can you tell me about it? I think transformation is only one part of the problem. The tougher problem is searching: First you need to find the code you want to transform. We found a pattern matching language either not powerful enough or to complex for a human being to understand (or both). We had a rather simple clean up example: 'Replace all write accesses to method parameters with local variables' I.e. given something like: public void foo(int i) { i+= 10; sysout(i); } would be replaced by: public void foo(int i) { int j= i + 10; sysout(j); } Can your language express such a search? That a variable, which is a method parameter, is written? Just saying that the parameter is on the left side of a '=' is not enough: There are other ways to write to a variable (i.e. ++) Other problems we had with pattern languages are expressing of non existence. I.e: 'Replace all empty catch bodies with 'JavaPlugin.log(${exception})'' 'Move all ${InputStream}#close() which are not executed inside a 'finally' block' into a finally block' ... Do you have a website with a set of such small examples and how you would implement them in your language? > I don't know if the Eclipse project is interested in something like that. I am certainly interested in such a functionality, but if we integrate it into JDT it would require to be "easy" to use for a large amount of users.
(In reply to comment #6) > > I don't know if the Eclipse project is interested in something like that. > > I am certainly interested in such a functionality, but if we integrate it into > JDT it would require to be "easy" to use for a large amount of users. I'm not sure how this could be very "easy" for users: programming is hard, and programming a program to program is harder :) Personally, I would consider Scala for this * it has pattern matching and given AST types that are usable as patterns, it should be relatively easy to program the transformations * it is fully interoperable with Java and about as performant * probably even more performant for recursion in some cases as it has more fitting data types (for example immutable list with cons) But supporting full Scala would probably require a Scala IDE so I guess it's not an option... I'm just letting my imagination run wild here, but here's another way it could be done: In Scala it is also easy to create parsers with parser combinators. Perhaps a simplified pattern matching language (an external DSL if you will) and a parser for it could be created in Scala. The DSL should be equivalent or easily translatable into a subset of Scala that supports enough of the Scala syntax to do pattern matching against a Java AST and define the transformations. The output of the parser would be a translation of the original script that is actually valid Scala code and is then compiled with the real Scala compiler. Hmm... perhaps this sounds too crazy?
(In reply to comment #6) > 'Replace all write accesses to method parameters with local variables' > I.e. given something like: > public void foo(int i) { i+= 10; sysout(i); } > would be replaced by: > public void foo(int i) { int j= i + 10; sysout(j); } > Can your language express such a search? That a variable, which is a method > parameter, is written? Just saying that the parameter is on the left side of a > '=' is not enough: There are other ways to write to a variable (i.e. ++) > Other problems we had with pattern languages are expressing of non existence. > I.e: > 'Replace all empty catch bodies with 'JavaPlugin.log(${exception})'' > 'Move all ${InputStream}#close() which are not executed inside a 'finally' > block' into a finally block' > ... No, AJaTS cannot express this kind of search.. > Do you have a website with a set of such small examples and how you would > implement them in your language? There are some examples at http://www.cin.ufpe.br/~jats/examples.htm e http://www.cin.ufpe.br/~jats/ajats/examples.htm . I´m convinced that this language is really not what you´re looking for. This kind of search and pattern matching cannot be expressed in AJaTS.
(In reply to comment #7) > I'm not sure how this could be very "easy" for users: programming is hard, and > programming a program to program is harder :) I couldn´t agree more. I can´t think of a easy way to do that. > Personally, I would consider Scala for this > * it has pattern matching and given AST types that are usable as patterns, it > should be relatively easy to program the transformations > * it is fully interoperable with Java and about as performant > * probably even more performant for recursion in some cases as it has more > fitting data types (for example immutable list with cons) Are you talking about http://www.scala-lang.org/?
(In reply to comment #9) > Are you talking about http://www.scala-lang.org/? > Yes, that Scala. I'll give a few more links: Using Scala to Explore Trees http://www.artima.com/forums/flat.jsp?forum=276&thread=218997 There's a PDF about Scala's parser combinators: http://www.cs.kuleuven.be/publicaties/rapporten/cw/CW491.pdf Some links to various documentation can be found on the Scala website, but a lot of it seems to be academic work, so maybe not the most readable. Daniel Spiewak has written a good introduction "Scala for Java Refugees": http://www.codecommit.com/blog/scala/roundup-scala-for-java-refugees
Created attachment 92172 [details] ast search (In reply to comment #8) > I´m convinced that this language is really not what you´re looking for. This > kind of search and pattern matching cannot be expressed in AJaTS. Mmm, yes, this looks more like an (advanced) text template mechanism. I even claim, that it is possible to implement most of this with our text template infrastructure (prove me wrong;-). (In reply to comment #7) > > > I don't know if the Eclipse project is interested in something like that. > > > > I am certainly interested in such a functionality, but if we integrate it into > > JDT it would require to be "easy" to use for a large amount of users. > > I'm not sure how this could be very "easy" for users: programming is hard, and > programming a program to program is harder :) Of course, that's why we are looking for a way to build such queries in a UI rather then to program this queries in a language like Scala, and hey, why not use Java? Our Java AST and ASTRewrite is pretty cool stuff too. > Personally, I would consider Scala for this > .... > Hmm... perhaps this sounds too crazy? Yes, and no, I have a "working" prototype which allows to build queries in a UI. The UI generates code for a functional language (the eclipse query language;-) and an interpreter can interpreter this language. The language operates on (any kind) of DOM (i.e. Java code ASTs). See screenshot. When I have time after 3.4 I want to make it available as plugin. But it's a long way from a quick prototype "hack" to a product quality feature. What we certainly don't want is to implement/integrate a Scala development tooling.
Created attachment 92175 [details] screen shot Here is a screen-shot from a more recent implementation.
(In reply to comment #11) > Of course, that's why we are looking for a way to build such queries in a UI > rather then to program this queries in a language like Scala, and hey, why not > use Java? Our Java AST and ASTRewrite is pretty cool stuff too. Yes, they are pretty cool, I've used them a little in the past. But why not make code written against ASTRewrite easier to plug-in -- for example by making refactorings written in a workspace project available through the UI, somewhat like Eclipse Monkey makes scripts available. The UI looks usable, but IMHO, it ends up being very similar to actual programming, just done using forms instead of a text editor. I think an autocompleting and highlighting editor should work as well (or maybe that's harder to write than those forms?). Anyway, I think the kinds of programmers who would be using this don't need that much hand-holding as the form seems to provide. But that's just my opinion. I'm mostly speaking as a potential user: I would want to use something like this if it were made easier. In the past have written some problem detection/quick fixes (if I remember correctly it was using the TPTP static analysis stuff & AST Rewrite), and I would do that more often, but currently it's too much plumbing to get them working. All I would want is to write the refactorings/static analysis in the same workspace where my code is and run them right away after writing (with minimal hassle). I don't really care that it should be done through an UI instead of programming them (but it's a plus if the programming would be at a similar level of abstraction as your UI) > What we certainly don't want is to implement/integrate a Scala development > tooling. Yes, of course. I never meant to suggest otherwise.
PS. If anyone reading this is still interested in using Scala for something like this (even if it wouldn't be able to go into JDT), take a look here. A blogger has already experimented with using Scala with Eclipse Java AST & ASTRewrite: http://langexplr.blogspot.com/2007/07/creating-java-refactorings-with-scala_24.html There's a lot of code actually, but probably a lot of it could be common for many refactorings, and the interesting parts would be quite easy to program using pattern matching, here's a snippet (and surely this could be simplified further): def propagateNegationInCondition(condition : Expression, rewrite : ASTRewrite, ast : AST) : Expression = { condition match { case EqualsExpression(x,y) => Utilities.createInfixExpression( rewrite.createMoveTarget(x).asInstanceOf[Expression], rewrite.createMoveTarget(y).asInstanceOf[Expression], InfixExpression.Operator.NOT_EQUALS, ast) case NotEqualsExpression(x,y) => Utilities.createInfixExpression( rewrite.createMoveTarget(x).asInstanceOf[Expression], rewrite.createMoveTarget(y).asInstanceOf[Expression], InfixExpression.Operator.EQUALS, ast) ....
(In reply to comment #13) > The UI looks usable, but IMHO, it ends up being very similar to actual > programming, just done using forms instead of a text editor. I think an > autocompleting and highlighting editor should work as well (or maybe that's > harder to write than those forms?). Anyway, I think the kinds of programmers Yes, absolutely, that's a very good point. One advantage of the UI is that we can hide the generated language as a implementation detail. This would allow us to change the language in the future if necessary. You can even imagine that we generate code for different languages (think Scala:-). Using an editor and all this (quick fix, refactorings...) is much more work. And it would require us to spec the language in detail. But adding an editor later is of course an option (maybe as separate plugin). > who would be using this don't need that much hand-holding as the form seems to > provide. But that's just my opinion. Yes, at the moment the UI allows to express everything, maybe, to make it simpler to use, the expressiveness of the UI must be reduced. > I'm mostly speaking as a potential user: I would want to use something like > this if it were made easier. In the past have written some problem > detection/quick fixes (if I remember correctly it was using the TPTP static > analysis stuff & AST Rewrite), and I would do that more often, but currently > it's too much plumbing to get them working. > All I would want is to write the refactorings/static analysis in the same > workspace where my code is and run them right away after writing (with minimal > hassle). I don't really care that it should be done through an UI instead of > programming them (but it's a plus if the programming would be at a similar > level of abstraction as your UI) Yes, that's exactly the idea behind all this. Static analysis is a good use case, code measuring another (i.e. how many types with @since 3.4 tag in jdt/ui).
Is there any part of this feature that could be done in three months, as a GSoC project?
(In reply to comment #16) > Is there any part of this feature that could be done in three months, as a GSoC > project? I think it's possible (but difficult) to implement/polish the AST search in that time frame based on my previous work. But the question remains if this is really what users want/expect? (Of course I think it is the coolest feature ever:-) Which part do you refer to? How would that look like?
Hi Everyone! This is the first time I post on bugs.eclipse.org. I've just sent my GSoC applicaction titled" Support for user-defined refactorings in JDT, slightly modified" with google's gsoc webapp. Here is the abstract : "Sophisticated code analysis and transformations can be performed using Prolog Inference Engine and Prolog rules. While traversing AST with an AST Visitor we can produce Prolog facts about examined source code. Anti-patterns, bad-code smells and design-patterns can be expressed as Prolog rules. Transformations of code can also be expressed as Prolog rules that would invoke AST Rewrite components to manipulate source code. For the Prolog part of the solution, an embedded "tuProlog" or "Gnu Prolog for Java" would be used. Prolog enables use of custom languages with its Definite Clause Grammars (could be though of kind of a father of DSLs). It would enable us to create a query language suitable for source code search and modification. This query language would be converted into Prolog rules and run by the Prolog Engine. In case our query language wasn't able to handle more complex situations, more advanced user could write queries directly in pure Prolog. Managing rules and running them would be aided with appropriate Eclipse UI." The pattern language would be expressed in DCG so typical user wouldn't be even aware he/she is writing in fact Prolog rules. Rules engine would be Prolog Ineference Engine that reasons about code or applies transformations. Usage of Prolog won't require any additional dependencies, IDEs nor software - it will be embedded into the package. (Anyway Prolog syntax is very straightforward, thus colored editors and hints are easy to implement) A hypothetical pattern language could look like this: DECLARE MakeIntStaticFieldsAsPublicFinal($packageName) AS MATCH STATIC $field, TYPE $field IS "int", FIELD $field OF $class, CLASSS $class OF PACKAGE $p OR INTERFACE $class OF PACKAGE $p, PACKAGE $p, NAME $p IS $packageName TRANSFORM MAKE PUBLIC $field, MAKE FINAL $field; and it would be automatically translated behind the scences into Prolog rules: makeIntStaticFieldsAsPublicFinal(PackageName) :- isStatic(Field), typeOf(Field,"int"), fieldOf(Field,Class), classOf(Class,P),package(P),nameOf(P,PackageName), makePublic(Field), makeFinal(Field). MakeIntStaticFieldsAsPublicFinal(PackageName) :- isStatic(Field), typeOf(Field,"int"), fieldOf(Field,Class), interfaceOf(Class,P),isPackage(P),nameOf(P,PackageName), makePublic(Field), makeFinal(Field). In case DCGs were too hard to implement with both tuProlog and Gnu Prolog for Java, JavaCC or similar parser-generator tool can be used to translate query language into Prolog rules. For example, the "behind the scenes rules" generated for "Replace all empty catch bodies with 'JavaPlugin.log(${exception})'" could look like : ReplaceAllEmptyCatchBodiesWithLog(PackageName) :- isEmpty(CatchBlockBody), catchBlock(CatchBlock,ExceptionType,ExceptionVar,CatchBlockBody), catchBlockOf(CatchBlock,Method), methodOf(Method,Class),classOf(Class,P), isPackage(P),nameOf(P,PackageName), replace(CatchBlockBody,join(join("JavaPlugin.log(",ExceptionVar),");")). ReplaceAllEmptyCatchBodiesWithLog("org.my.package"). Executing theese rules should insert the string into all empty catch blocks within org.my.package.
(In reply to comment #18) Hi Jakub Have you had a look at JQuery: http://jquery.ca/ ? What you are proposing sounds like an extension of JQuery. My conclusion from the JQuery evaluation is: "For me it is clear, that, although the most elegant solution, JQuery is not an option here: It is way too hard to understand. Prolog is a language that most programmers only know from classes back at university. Also a working plugin for JQuery exists now for a couple of years. It might be due to the sometimes strange UI behavior why this plugin never got accepted by a wide range of developers but I think the reasons are deeper, otherwise someone would have fixed the (minor) UI problems. TyRuBa is for sure a nice prolog interpreter, but I doubt that the average user is willing to learn all the details required to write a new predicate." But this is only my opinion. And the evaluation has been made about 2 years ago. It might be worth to reevaluate JQuery. Also translating a pattern language to prolog my ease the prolog pain.
Thanks Benno, I hadn't seen it before, It will be a great help for my Master Thesis. I've had a look at JQuery and it seems indeed my aprach is a kind of an enchancement to that solution. The question now is if it is the aproach that fulfills to this Eclipse enchancement request. That is, is it a right way to implement pattern language and rules engine that would be enbedded in JDT that Ed wrote about? I know the drawback might be performance here, but with Eclipse Jobs API we could avoid freezing UI by doing search/transformation asychronously in the background. JQuery haven't met wide adoption and there must be important cause behind it. Reasons for it might include: - JQuery in its current shape does only search. And default query set is still very basic. Why should I learn new GUI and query language to e.g. match all static-private fields since I can run a simple "Search in text" action? The amount of effort to get through a list of matches and correcting code manually is lesser than mastering this new tool. I think if it were enchenced with transform abilities and more complex queries in deafult installation, enclosed in a simple and intuitive UI, benefits could outwight the cost of learning curve. - To perform more complex search, JQuery needs a lot of user input (writing queries and lots of clicking that distracts user from what he/whe was doing). In contrast, popular tools jak PMD and checkstyle require minimum user input (just few clicks) to get a list of matches. - The idea of quering code is quite fresh to typical developers. Few have heard about Jackpot or JQuery. People use QuickFix because it is integrated in JDT but many people haven't ever used TPTP yet, for instance. People tend to stick to box-delivered functionality and seek for plugins when they need them. This idea is fresh and I think few people realize It can be done and what they could benefit from it. Another question is what a query langauge should look like? The goal of such a language would be to hide complexity from the user and let him/her express what they want easily. That implies the language might prevent the user from making complex queries. That's why I think possibility to use Prolog rules directly still matters. And there are Erlang and Haskell developers out there using Eclipse for Java, C++, Erlang and Haskell. Since Prolog is somehow similar to functional languages (recursive programming) I think they might be interested to write new queries. How should such a langauge look like? Should it be a verbose one like, say, SQL? Or should it be more pattern-oriented, like some exemplary Jackpot rules?
I think transformations should be pattern based. Everybody understands patterns. If you use prolog or scala or Java or some other language under the covers where the user doesn't see it, that's fine, but the part exposed to the user needs to be very simple or it won't be used very much. For a SoC project why not just port Jackpot to Eclipse? The first phase would be just to make it work. Second, to isolate the Eclipse/Netbeans differences to one module (which might use the compiler API JSR implemented by both) and contribute back to Jackpot. Third, reimplement all current Eclipse refactorings in the new system, extending it as necessary. If current refactorings can be expressed much more succinctly in this system than their current Java implementation then you have a winner. On the other hand if it's just as hard or harder to do them in the new system then that means something is wrong and more work is needed, perhaps to simplify it further or codify common patterns and idioms (i.e., reduce redundancy or verbosity in the refactoring expressions).
Hi Ed! What exactly do you mean by pattern language? Is it meant to resemble actual code of refactored language? Should it look like : { catch( Exception $e ){ } } => { catch( Exception $e ){ JavaPlugin.log(${e}); } } Is it possible to parse a piece of code instead of an entire CompulationUnit into an AST tree with JDT facilities? As for GSoC there might be a problem if Jackpot isn't released open-srouce early enough. They've decided to do so but it is not clear when it's gonna happen. And how about JQuery approach? It has also other use-cases, for instance aiding with documenting existing frameworks - the ability to extract design patterns would help in understanding concepts embodied within projects. These issues are often overlooked in documentation - docs tend to include javadocs and some comments, guidelines but not necessarily low level details of design itself.
i very much like the idea proposed here. maybe it is time to do this in e4!?
