Community
Participate
Working Groups
This work involves creating a new CDT parser for C99. The parser will be generated from grammar files using an LALR parser generator named LPG. LPG is a very easy-to-use and feature rich parser generator. It supports automatic recovery from syntax errors as well as a backtracking parser for handling ambiguous grammars. Automatic AST generation is also supported; however we do not plan to use this feature since CDT already contains a C99 compatible AST that can be easily reused. The grammar files will contain calls to actions that are used to generate the AST. This work forms the basis for a framework that would allow CDT to be extended with different flavors of the C language. LPG has a very handy feature: it allows one grammar file to include another. The including file may add and redefine grammar rules as necessary. This feature can be used to add rules for language extensions and generate new parsers. LPG on sourceforge: http://sourceforge.net/projects/lpg/
Created attachment 58359 [details] preliminary plug-in This is a preliminary plug-in to give people an idea of how this work is progressing. Two grammar files are provided so far, one for the scanner and one for the parser. The preprocessor has not been implemented yet. A small test suite is provided that extends from AST2Tests. The LPG java runtime is not included. It can be downloaded from sourceforge. The file lpgjavaruntime.jar should go in the root of the plug-in. Any questions or comments are appreciated…
Created attachment 58360 [details] parser tests patch This patch adds constructor methods to several of the parser test suites. This allows the current test suites to be reused with the C99 parser.
LPG 1.1 has been added to the orbit project. We are refactoring the plugin to consume LPG from there.
I reworked the plugin to use the LPG from Orbit. I've not included the JAR... we have to figure out how all this is going to fit into the CDT build process. For now people that want to try the plugin out will have to get LPG from the Orbit download page. I've refactored the plugin so that the tests are in their own plugin as well. Everything lives in CVS under org.eclipse.cdt/c99. It should all show up shortly. I'll leave this Bugzilla open to facilitate further patch drops on the feature. Thanks Mike, keep up the good work!
At the end of the day, we need to use the LPG plugin as it sits in Orbit. This will allow the CDT and other projects in Europa and beyond to use the same copy. We already do that of sorts with our test plugins which use a couple of plugins out of the eclipse repository.
(In reply to comment #5) > At the end of the day, we need to use the LPG plugin as it sits in Orbit. This > will allow the CDT and other projects in Europa and beyond to use the same copy. > We already do that of sorts with our test plugins which use a couple of plugins > out of the eclipse repository. Agreed. We just need to figure out the logistics. Really all the Orbit stuff should just be on the Europa update site and we could reference a hypothetical feature for LPG that was on that site. They don't currently have an update site though.
Agreed. I was just concerned that you were thinking of including the LPG jar in your feature. It should be in its own feature.
One other thing that I didn't mention that I wanted to get opinions on... The current GNU parser isn't really maintained much... most of the original authors have moved on to other things, and it's hard to navigate the depths of that code. I'm thinking at some point down the road that it would write a GNU extension on top of this C99 parser and then replace the old parser with it. Then it would theoretically be easier to fix bugs as well as to cope with changes to the language over time. Nothing imminent (definitely not for 4.0), but food for thought.
(In reply to comment #8) > I'm thinking at some point down the road that it would write a GNU > extension on top of this C99 parser and then replace the old parser with it. Of course, before doing that you'd make sure that all the JUnits pass and the indexing performance isn't severely degraded ;).
(In reply to comment #7) > Agreed. I was just concerned that you were thinking of including the LPG jar in > your feature. It should be in its own feature. Hmm... From the Orbit Wiki: "There are no features from Orbit, for consumers to use. The goal is that these bundles can be included in other project's features as needed. While this results in some duplication in projects zips or update sites, since all projects use exactly the same bundles from Orbit, there should be no duplication once installed." Looks like we'll have to bundle it ourselves.
Did you check with Jeff McAffer to see if this is really the policy? Seems to me that if you have to do your own IP check and releng, what the hell would the point be then...
(In reply to comment #11) > Did you check with Jeff McAffer to see if this is really the policy? Seems to > me that if you have to do your own IP check and releng, what the hell would the > point be then... I guess consistent bundling? I'll talk to him.
Created attachment 60810 [details] preprocessor contribution This patch adds a new preprocessor implementation to the C99 parser. This preprocessor is token-based, and sits between the lexer and the parser. Several test suites are included, with more to follow. Most of the C99 parser so far is brand new. However the AST implementation and the LocationMap class from CDT are being reused. I would like to further modularize the preprocessor in the future in order to make it reusable with other languages. I remember reading something about how the fortran people want a C preprocessor. This patch includes a slight modification to the core tests to allow some of the test cases to be reused. Moving on to implement content assist :o
I was trying to upload the patch yesterday but bugzilla kept giving me mysql errors, thats why there's four empty patches in the attachment list. I couldn't figure out how to get rid of them so I just marked them obsolete.
Created attachment 60842 [details] updated preprocessor patch Updated patch, marked failing test case.
(In reply to comment #15) > Created an attachment (id=60842) > updated preprocessor patch > Updated patch, marked failing test case. Applied to HEAD. Thanks, keep 'em coming :-)
Created attachment 62793 [details] patch number 3 This updated patch adds: - content assist support - much better error reporting in the preprocessor - better integration with CDT - more test cases
I'm noticing some missing NON-NLS tags and some unused imports but other than that it looks good. Once I get signoff on committing 179393 I'll commit this along with it. I'll clean up the NLS tags and imports so don't bother submitting another patch to fix those.
Applied the latest patches. Thanks Mike.
Created attachment 65133 [details] Patch #4 (attached on behalf of Mike) Mike has forwarded me patch #4. For the sake of saving time I'm attaching this on his behalf. I'll be committing it shortly
Applying Mike's latest patch to HEAD. This feature is essentially feature complete. I'm going to mark this as FIXED. Any outstanding bugs we'll track separately with new Bugzillas.
BTW there was a slight problem in that there was a duplicate build path entry on the tests plugin. I fixed that while I was at it.
