Regarding 1, who cares? We already know how to parse source code
into an AST. Moreover, if you ever want to read the source code, you
would need to translate the AST back into source, where, presumably,
someone would edit the text, at which point you would need to parse to
build the AST again. So persisting an AST doesn't free you from the
need to parse.
It is true that parsing would still be necessary in many cases for
editing. But it is more logical to parse when the user makes a change.
Regarding 2, I'll admit that dynamically redefining grammars in
specific context sounds like an interesting idea. But what problem
does it solve? How does Maya let one express something more elegantly
or efficiently when compared to vanilla java? Perhaps more to the
point, it sounds like the new parsing techniques necessary to make
Maya work were already figured out, while it isn't at all clear that
the user-interface for tools operating direcly on an AST is
well-defined. And if the user ever changes source code, you would
need to parse to build your AST. So you still have to write a parser.
The paper about Maya gave a few interesting examples. One was the
foreach construct (which is now part of Java). Another was multimethods
and open classes, which is quite a big evolution from the base language:
multimethods permit to select which implementation of a method is called
at runtime according to the actual type of all its parameters, in
contrast to java where only the first, implicit parameter (this) is used
for method selection. Open classes permits to define methods outside of
a class.
An interesting property of Maya is that it lets the programmer apply
language extensions to selected portions of the code. This can be used
as an aspect weaving technique, with the added benefit to be able to
extend your syntax if needed.
Maya itself is a powerful substrate for implementing new languages
(that's also what LDT aims to be). But it lacks a very important part of
language development: the associated tooling.
As far as tools for operating directly on ASTs are concerned, I think
there are several existing interaction techniques:
- a visual class diagram editor. It is be easier to implement if based
on ASTs rather than text
- a syntax aware text editor (like JDT's). It is conceptually such as tool.
Well, these are two straightforward examples, and they are already
implemented in some products. Having a full AST manipulation framework
would enable the creation of many more of these tools.
But right now with Eclipse if someone wants to implement an editor based
on the AST instead of the text representation he has to pay a lot of
attention to proper update of the source code in parallel with AST
modifications. You cannot forget the text.
Regarding 3, this common substrate is what the LDT is going to try to
do. But the plan is to address it at an API level. Not a serialized
on-disk format.
I think there is a compromise (see below)
Regarding 4, I think having a high-level language to express how
tooling should be generated is a good and interesting idea. However,
I'm not clear on how persisting ASTs forces this to happen.
Ok, maybe I exagerated a bit saying that it would force us ;o) But it
would greatly facilitate it.
By the way, the article on Jetbrains mentioned by Chris is a really
excelent read on this matter.
Now, lets consider what you would lose if you were to persist ASTs.
First, you would be forcing users to either use the eclipse toolset,
or to understand the serialized AST format. If someone wants to read
the persisted format outside of eclipse, they need to understand this
format, and I don't think it is fair to ask people to do that. So
think about how frustrated people will be when the want to use another
IDE, or emacs/vi/notepad. Also, what about diff tools? Source
control tools? Command-line text-processing tools such as perl, wc,
grep,...? What about integrating projects into pre-existing build
systems?
These are very strong and valid counter-arguments. I personally have no
easy solution for them, apart from developing a new SCM system suitable
for keeping the history of a source code database...
So maybe a good compromise would be to extend the AST API in Eclipse so
that:
- it is feasible to create a tool that reasons *only* on ASTs
- a plugin could be developed that provides AST-level storage and SCM
(meaning filesystem storage would be another plugin)
Regards,
Guillaume