[recommenders-dev] JFace Template Grammar: A big refactoring and next st

[Andreas Sewe <andreas.sewe@xxxxxxxxxxxxxx>]

Hi all,

during a discussion with some of the Xtext developers at EclipseCon
Europe about our experience generating an editor for the JFace template
language, I got the recommendation to restructure the grammar such that
it focuses on the embedded language (i.e., JFace's ${} variables) rather
on the surrounding language (i.e., full-fledged Java).

Here's why I think this is good advice: Writing a grammar that supports
Java++ (aka Java + ${} variables) in one go is a daunting task, in
particular as there are legitimate use cases for snippets that are not
valid Java statements or expression at all. This built-in template, for
example, is not a valid Java expression:

  else if (${condition:var(boolean)}) {
      ${cursor}
  }

Following the advice, you would just parse the above as a sequence of
Java tokens ("else", "if", "(", ...) interspersed by stuff from the
embedded language. This is a much more manageable task.

And the constructs of the embedded language can parsed "in detail";
everything between the "${" and "}" follows a structured grammar.

Using this approach you can get code completion of all kinds of ${}
constructs and cross references between IDs defined therein. You won't
get a useful outline, but in the light of Java fragments like the above
"else if" snippet, it is doubtful whether this is doable at all. You
can, however, improve support for the outer language (Java) by, e.g.,
adding heuristics to make sure that parentheses are balanced (even a
snippet will probably close every "{" with a "}").

Now, how to best proceed from here?

I rewrote the grammar [1], which resulted in a lot of breakage in code
that assumed the old AST structure. So I had to (temporarily) dump that
code. I also removed most of the MWE2 fragments, so that we start with a
fairly minimal feature set, as I think proceeding one feature at a time
is the best way forward.

And the feature(s) that are currently most important are IMHO the
following two:

1.) make sure that the template language's grammar is correct, i.e.,
that at least all the snippets built into Eclipse work (show not parse
errors)

2.) Get some amount of syntax highlighting to work. This should be more
than just keywords and show us that the parser understands the grammar
as we do.

Step 3 would then integrate Content Assist again. (Here, Stephan had
already made great progess and I am sure we can incorporate much of what
he has learned/done.)

Anyway, you can checkout the change from Gerrit [1] and build it using

  mvn clean install -pl :org.eclipse.recommenders.templates.tests -am

This should still show you a couple of grammar errors (8 out of 30 at
the moment), but there's definitely progress.

Best wishes,

Andreas

[1] <https://git.eclipse.org/r/#/c/18168/>

-- 
Codetrails UG (haftungsbeschränkt)
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Mobile: +49-170-811-3791
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940