Community
Participate
Working Groups
Created attachment 102024 [details] first draft Summary The Eclipse Java Development Tools (JDT) excels at supporting the editing and navigation of Java code, setting the bar for newer IDEs, including those for Domain Specific Languages (DSLs). Although IDE generation keeps making progress, entry barriers remain high, thus forcing many developers to rely on traditional ways to encapsulate new language abstractions: frameworks and XML dialects. We explore an alternative path, Internal DSLs, by automating the generation of the required APIs from Ecore models describing the abstract syntax of the DSLs in question. To evaluate the approach, we present a case study (statecharts) and discuss the pros and cons with respect to other approaches. Most embedded DSLs, while offering a user-friendly syntax, are fragile in the sense that their expressions may not comply with the full static semantics of the DSL in question. Productivity studies recommend that errors should be reported while the frame of mind is still focused in the error location. To address this issue, we leverage the extension capability of Eclipse to detect at compile-time malformed DSLs expressions. The technique relies on mainstream components only: EMF, OCL, and JDT. We conclude by previewing ongoing work aimed at improving the support for embedded DSLs by performing language processing as a background task. The prototype described in this article (DSL2JDT) has been contributed to EMFT and is available from CVS as described in the Source Code section below. By Miguel Garcia, TUHH (Technische Universität Hamburg-Harburg, Germany)
Created attachment 102172 [details] added a Table of Contents and "Instructions for the Impatient" section
This is really interesting :)
Very cool!
Created attachment 102883 [details] more detailed explanation (figures, examples) of generated progressive interfaces Guys, I know this technology is cool, otherwise I wouldn't have written an article about it ;-) OK, now seriously. If you want to benefit for real from DSL2JST, please let it loose on some large .ecore model and let us know how you evaluate the results, so as to find areas for improvement in DSL2JDT. For example, the very latest CVS version of DSL2JST now handles .ecore models declaring java.math.BigDecimal (that was brought to my attention by the SQL metamodel in the Data Tools project). And so on so forth. keep up the good work, Miguel
Cédric, Boris: do you have any specific comments for Miguel? Anything in particular he should focus his attention on? Or do you feel that the article is ready for publishing?
I need some time to read the article in detail. Miguel, where do you want feedback on the technology itself (not the article)? Newsgroup? Mailing list? Bugzilla? (Specific answers including e.g. the name of the mailing list would be good.)
(In reply to comment #6) Boris, To keep the discussion focused, I'd prefer comments and patches for DSL2JDT in the form of bugzilla entries, under Modeling > EMFT > Emfatic. Because of historical reasons, DSL2JDT can be found there. The technology can be expanded in many dimensions, some of them related to your background. However, I would make a priority to correct the examples, slightly improve the article (and thus get it out the door), and only then move on to substantive case studies (GUI description languages, databinding) or full DSL-aware language processing. That's enough material for another article, and I'd be happy to help review it! (as long as you author it :) Miguel
Created attachment 103036 [details] IRex Iliad-formatted PDF of the article
I printed this yesterday and also formatted it for my Iliad e-book reader. The graphics are too wide to display on US Letter or European A4 paper, much less an e-book reader like the Iliad (about A5 paper size), without clipping. I suggest that in the interest of printability and portability that the graphics be rearranged so at least they can be printed on US Letter and/or A4 paper without clipping.
(In reply to comment #9) Your wish will be granted! (in the next version). Miguel
(In reply to comment #9) > I printed this yesterday and also formatted it for my Iliad e-book reader. Any comments about the content?
(In reply to comment #11) > (In reply to comment #9) > > I printed this yesterday and also formatted it for my Iliad e-book reader. > > Any comments about the content? To be frank, I've now read it twice and think I've got the gist of it. But I'm having trouble understanding how to apply it to my work. I tend to be in the camp that if I want an embedded DSL, I'll use Scala or JRuby so that I can get away from Java's ugly receiver.message(args) syntax. The result is just much more readable. So part of what I'm trying to understand is when a Java-based DSL would be useful. I admit that as soon as I see a JMock example, I'm automatically biased against what I'm reading. I hate JMock, mostly because it embeds method names in strings, so you can't refactor JMock-tested code without breaking all your unit tests. I call it write-once testing. And I don't find it to be particularly intentional when I come back to it later--ie: I find it hard to discern what my intent was when I wrote the test to begin with. Since JMock is my only prior experience with a Java-based embedded DSL, I need to overcome a bit of personal bias here. At this point, probably the most constructive suggestion I could imagine is for the author to pick an example from a problem domain that is related to business application development. The state chart example feels contrived to me, since I almost never imagine myself programming a state machine directly (except if I'm implementing a communications protocol, but not many people do that very often any more). But basically I think I need some more time to digest this and see if I come around to the author's point of view. Or some subset of it. :-)
(In reply to comment #12) Dave, Thanks for reading the article draft (twice!) ;-) I guess I can take care of the jMock example by being open about its shortcomings, specially its fragility upon refactoring (the same would have been the case with an Scala implementation). More in general, I didn't make clear for any of the existing, "manually-built" DSLs what well-formedness rules (WFRs) they should obey. Additionally, in the case of jMock those WFRs also encompass Java AST nodes (i.e., nodes for the methods declarations referred to in the jMock snippets). jMock WFRs turn out to be more complex than I initially thought. On the other hand, the WFRs of statecharts are self-contained, but statecharts are not representative of business app development. I must agree: no EJB container goes all the way to require the protocol of a session bean (e.g. the allowed invocation sequences on a shopping cart). But that's because ... DSLs are not easy to embed in Java without DSL2JDT! Coming back to statecharts, suggestions are welcome about a not-so-complex, business-development-oriented DSL with well-formedness rules (to bring to bear their compile-time checking) that at the same time may office as introductory example (that's what the statecharts example is). Most of the business development DSLs I know are XML-based, and that makes more difficult to dig out from manuals their well-formedness rules (those not captured by the XML Schema grammar). If staying with the XML syntax instead of its Java-embedded counterpart, then tools like SmartEMF (briefly described in the article) are better suited for that task. Don't make me study an XML case study, please! ;-) Miguel
Created attachment 103265 [details] Minor corrections list
(In reply to comment #13) > (In reply to comment #12) > Thanks for reading the article draft (twice!) ;-) :-) > I guess I can take care of the jMock example by being open about its > shortcomings, specially its fragility upon refactoring (the same would have > been the case with an Scala implementation). :-) Yes. > On the other hand, the WFRs of statecharts are self-contained, but statecharts > are not representative of business app development. I must agree: no EJB > container goes all the way to require the protocol of a session bean (e.g. the > allowed invocation sequences on a shopping cart). But that's because ... DSLs > are not easy to embed in Java without DSL2JDT! > > Coming back to statecharts, suggestions are welcome about a not-so-complex, > business-development-oriented DSL with well-formedness rules (to bring to bear > their compile-time checking) that at the same time may office as introductory > example (that's what the statecharts example is). Most of the business > development DSLs I know are XML-based, and that makes more difficult to dig out > from manuals their well-formedness rules (those not captured by the XML Schema > grammar). If staying with the XML syntax instead of its Java-embedded > counterpart, then tools like SmartEMF (briefly described in the article) are > better suited for that task. How about duplicating (part of) Glimmer as a Java-based internal DSL? --In a step-by-step fashion--so I call follow what you did, and more importantly, why you did it? Dave
(In reply to comment #15) Dave, DSL2JDT started its life as a component in a toolset for language engineering. This bottom-up approach shows in the (current) lack of a large-scale case study, of the likes of BPEL or JPQL or Glimmer (the same applies to external DSL generators which have been around for a much longer time). My plan at this point is to entice early adopters to apply DSL2JDT to their DSL of choice, so that they report on their success :-) That line of work has already started at my university department, with two case studies: GUI description (no progress so far) and JPQL, the OO query language for JPA Persistence. The latter looks more promising at this point (I should update the article to reflect that), as follows. The JQPL metamodel originally documented in http://www.sts.tu-harburg.de/~mi.garcia/pubs/atem06/JPQLMM.pdf is being brought up to date for JPA 2.0, based on the JSR-317 Early Draft and unofficial information such as http://java.dzone.com/articles/looking-forward-jpa-20 http://java.dzone.com/articles/looking-forward-to-jpa-20-part A JPQL case study would be interesting because it could be benchmarked against all of: (a) the current practice, i.e. JQPL as string literals in Java code (b) custom editors such as those in WTP Dali and in commercial tools (c) manually-built DSL embeddings, like that for SQL (ok, not exactly JQPL) at www.jequel.de (d) if some generator of external DSL took up the gamut, we could also benchmark against them ;-) Problem is, none of those DSL2JDT-based embeddings (JPQL, GUI layout) is yet ready, and once they are, each will easily require a dedicated article. Coming back to Glimmer (that was a long introduction!) it's by far the DSL where I am less experienced. I'd get working a JPQL or BPEL embedding in much shorter time. Besides, (a) where is Glimmer (the DSL) defined? In source code. (b) where are the well-formedness rules of Glimmer defined? In source code. Summing up, the *language definition* of Glimmer is not explicitly stated, and recovering it from source code is much more difficult than picking an .ecore and .ocl already available (as for JPQL and BPEL). The situation around Glimmer reflects a best practice promoted by DSL2JDT: we achieve interoperability by asking DSL designers to jumpstart language design from an .ecore with validation rules. If Glimmer had it, I could try to do the rest.
The suggestion of Glimmer was merely so I would have a concrete example not done in Java for comparison. I'm happy with a subset of Glimmer or even something else. For example, Scala includes most SQL syntax as an internal DSL. The main point is to have something concrete -- SQL or UI or from a domain that RCP programmers will generally have experience in for comparison.
(In reply to comment #16) for those interested in embedding query languages in Java When I mentioned that a DSL2JDT-style embedding of JPQL could be compared with that for SQL, I overlooked a closer match for comparison, namely a LINQ implementation for Java at http://quaere.codehaus.org/ (whose API was developed "manually" and not generated from a .genmodel or grammar) Along the same lines, if you're familiar with XQuery you might want to try embedding it in Java. Its metamodel (XQuery.ecore) can be found at: http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gmt/AM3/org.eclipse.am3.zoos.atlantEcore/?root=Technology_Project
Some feedback on the article: == Overall idea The idea of embedding some declarative language into Java is very cool in general. Practically it's much better than manually using EMF factories. The biggest drawback is that it's still too verbose and thus not readable: I started using it for testing my code yesterday, but finally I decided to give it up and spend half an hour on building simple textual DSL using ANTLR. Probably it is a consequence of my problem domain nature: expressions of some regexp-like language. At least the approach is not suitable for all metamodels. I noticed that in many cases I would prefer constructors or factory methods with parameters and the following functions for building arrays and lists: public static <T> T[] $$(T... items) { return items; } public static <T> List<T> $(T... items) { return Arrays.asList(items); } (Generally I found '$' and '_' good tools for making some embedded languages in Java). == About well-formedness checking In addition to JUnit approach described in the paper, I would suggest using assertions embedded in generated builder code which check objects' well-formedness upon creation. I think this might be not so trivial to implement but probably it's worth thinking about it. == Implementation of in-place translation Most likely it's my fault, but I did not get the main idea: what is going to be translated to what :( == Some technical drawbacks I've found in DSL2JDT: 1. Boolean property setters generated by deafult (no additional annotations provided) do not take default values into consideration: so if I have an attribute which default to true, DSL2JDT gives me no way to set it to false. 2. Probably it's possible to avoid requirement of calling toAST() on each object, like this: sequenceQuery() .attributes() .definitions( sequenceWildcard().toAST(), symbolQuery().attributes( attributeQuery() .attributeValue(attributePresence().toAST()) .attributeName("a").toAST() ).toAST(), symbolQuery().toAST() ) .toAST(), I think it's possible to achieve using overloaded methods. It would be great to call toAST() once - at the end of the statement.
Miguel, as you asked me, here's a link to my article: DSL development based on target meta-models. Using AST transformations for automating semantic analysis in a textual DSL framework http://arxiv.org/abs/0801.1219
(In reply to comment #19) Thanks, Andrey. I have to go in more detail through your comments, but some quick notes follow: a) it's a bug for DSL2JDT to overlook the default of booleans. I'll also check what other defaults are being overlooked and fix that. b) now that you mention it, I see your point about the usefulness of symbols ($$, _) when defining an embedded DSL. Perhaps for those cases it makes sense to agree on an EAnnotation to convey that information to DSL2JDT, much like the existing EAnnotation for custom yes/no methods does. For example, on()/off() can be generated by DSL2JDT, with the net effect of being more readable than e.g. setIsOn(false) (BTW, although regexps and grammars are traditionally expressed with a plethora of symbols, others believe one can come out without them. JParsec provides some evidence on that, http://jparsec.codehaus.org/JParsec+Overview ) I guess the next version of the article will have more than a few udpates ...
(In reply to comment #18) another installment on the series "query language embedded in Java" Besides the fact that XQuery.ecore is available, I forgot to mention that type checking of embedded XQuery can also be done at compile time, by using the XQuery Normalizer and Static Analyzer tool (thanks, IBM Software Labs, India!). http://alphaworks.ibm.com/tech/xqnsta That Java library accepts an XQuery AST as input (POJO, would have been 100% ideal it it were EObject-based) and then: --- begin quote --- Given an XQuery expression, the tool uses the normalization rules specified in the W3C XQuery Formal Semantics document to convert the given expression to an expression in a core grammar (a subset of the XQuery grammar). ... Given a normalized expression, the tool again uses the static typing rules specified in the W3C XQuery Formal Semantics document to determine the output type of the expression. ... The Static Analyzer also checks for semantic errors (such as passing an empty expression to a function call where an integer argument is expected) --- end quote --- Now, how many *dedicated* XQuery editors type check at query-edit time? And that's not a toy DSL, either. I guess someone should try DSL2JDT on this one.
Created attachment 105176 [details] stylistic changes and clarified content up to and including JavaSwul example I've made some stylistic changes and clarified some content, based on the comments. At least one nasty bug has been fixed (cross-package references) yet others remain (the painful workspace-relative URIs and non-default default values :) i.e. the bug discovered by Andrey). The PDF also looks better (but for A4 only). Don't worry, I'm planning at least another round of improvements before claiming the draft to be ready.
Created attachment 105177 [details] PDF in A4
I find the idea quite useful, but only in some cases, when there is a real need to mix the DSL languages with java. It might be quite a conservative view, but the AST syntax makes code hard to understand. Nevertheless, interesting approach and article. Also like the statecharts example.
Generating Customized Code with DSL2JDT: DSL2JDT works fine with the following boolean properties of the genmodel, set to True (default is False for these properties). 1. Suppress EMF Types 2. Suppress Meta Data 3. Suppress Model Tags Initially, if the Model code is generated with the above properties as False, then before generating the Model code with these properties value set as True, the old generated packages have to be deleted first (for same base package name).
What is the status of this article? FWIW, I need the article in HTML or DocBook format to publish. We do not publish in PDF format.
Created attachment 108319 [details] article in html format Wayne, The article draft is doing fine. Several minor improvements have been made, both to the article and the tool (no more Ecore URI worries, for example). The article as it stands is ready for final review. There are pending feature requests for the tool (e.g., supporting EMF Generics and ecore models retrofitted from XSD). Such improvements will gain traction once developers know about the article's message. Additionally, the included pointers to related work are by themselves a valuable resource for those making their way through the DSL tooling jungle. I haven't found such discussion in other articles :) Miguel
Sorry for the delay. I'll start a review. Can you list the specific Eclipse projects and versions that are employed in the article?
(In reply to comment #29) Wayne, The plugin dependencies for the DSL2JDT tool are as follows (reproduced from the article draft): This article is known to apply to the following Eclipse projects: * Eclipse Platform, release 3.3 * Eclipse Modeling Framework, release 2.3
Hi, If you've been following this article draft you're probably on the DSL bandwagon. What about a face-to-face meeting at Eclipse Summit Europe? http://www.eclipsecon.org/summiteurope2008/ AFAIK, Birds-of-a-Feather are not being organized yet, but we might as well start thinking about that. In any case, it would be interesting to know what other DSL fans like you are doing or planning to. have fun, Miguel
I've copied the article to the eclipse.org CVS. :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse The directory can be found at: /www/articles/Article-AutomatingDSLEmbeddings You can see it online here: http://eclipse.org/articles/article.php?file=Article-AutomatingDSLEmbeddings/index.html Note that it hasn't been hooked into the resources database, so there's no link to it from the rest of the site (yet). I've changed the name of the HTML file to index.html and have modified it somewhat (the info box at the top of the page is automatically generated). We can overwrite this with new versions; if you'd like to submit a patch instead, make sure the patch is relative to the article directory. A quick read leads me to believe that the article is ready for final editing. Can some of the reviewers/commenters let me know how they feel about it?
Created attachment 110840 [details] mylyn/context/zip
Wayne, I've gone through article, looking for broken links and so on. I found that three images are not displayed, due to case mismatch between their file extensions (in the images folder) and how they are listed in index.html (the problem did not surface in my computer as I use Windows XP) The required changes in index.html are: on line 571, should read: images/manualconstraintB.PNG on line 568, should read: images/manualconstraintA.PNG on line 469, should read: images/ClassDeclEcore.PNG Other than that I've found no errors. (please don't ask why I don't send a patch, this time it was not XP's fault but mine)
(In reply to comment #34) > The required changes in index.html are: > > on line 571, should read: images/manualconstraintB.PNG > on line 568, should read: images/manualconstraintA.PNG > on line 469, should read: images/ClassDeclEcore.PNG Fixed.
Consternation. I forgot to correct the case of one more figure. On line 471. It should read, images/ClassDeclAPI.PNG (i.e, make file extension uppercase) I'm appalled at my oversight. But then, I'm a Windows XP user ...
(In reply to comment #36) Wayne, Just today I've submitted a paper (details below) on supporting the all-new LINQ query language in the context of ORM for EMF. And guess what? In that paper I'm citing this technical article, for its content about DSL embedding. It would therefore be great if my last typo (comment #36) were fixed and the draft published. I ask for neither Eclipse T-shirt nor fame, just letting others know about the virtues of DSL2JDT! Garcia M., Prithiviraj R. Rethinking the Architecture of O/R Mapping for EMF in terms of LINQ http://www.sts.tu-harburg.de/~mi.garcia/pubs/2008/ese/linq4emf.pdf Submitted to Eclipse Modeling Symposium at Eclipse Summit Europe 2008
Done. I had been waiting for a final +1 from a reviewer, but I'm going to accept the earlier generally positive feedback and absence of more recent negative feedback as acceptance. Send me your coordinates (mailing address, phone number) and shirt size and I'll send you some appreciation swag.
Marking this as FIXED. Will await your coordinates regardless.
