[cdt-dev] C dialect source editing

[Eugene Ostroukhov <eostroukhov@xxxxxxxxx>]

Dear CDT developers,

I spent last few weeks developing a CDT-based source editor for our language dialect. Our dialect is basically C++ with few new declaration specifiers (not unlike extern, const) and one custom construct for the cross-device function call. The goal was to make all CDT editing features support our language - and try to leverage all the great features you have created.

I have all the hacks we made to CDT published on GitHub - https://github.com/eugeneo/cdt/commits/customcdt8. I know they are ugly as I was hasty to create proof-of-concept so we know we can deliver a product we want. Now I am hoping to refine those changes and submit them to the upstream project.

I had to overcome following restrictions:

1. We needed a way to store arbitrary node attributes in the CDT index. This is vital for our code as most other features (highlighting, refactorings) depend on those attributes. https://github.com/eugeneo/cdt/commit/0263069415a07d1e2d46feaadfaff1a9dd9efbff, https://github.com/eugeneo/cdt/commit/03cd7cf4fecf181c81a4e5bc37666b9b9ea9c187 add ill-named "extendedBits" to some AST nodes. The idea was to have bit fields that CDT does not touch - all it does is makes sure they are always available from the nodes, persisted in the PDOM and are properly carried around. This feature is only required for custom languages support and is not visible to the vanilla CDT user.

2. We wanted to provide custom semantic and background highlight for our code based on the attributes I mentioned above. https://github.com/eugeneo/cdt/commit/49f94135fb2fc8c84c64276a142726e90f29f2dc and a few more changesets introduce an extension point that allows contributing semantic highlight rules, background highlight processors and nodes to the "Syntax Coloring" preference page tree. "Inactive code highlight" was refactored to leverage this framework (it is configurable through the "Syntax Coloring" page as well). We believe this would be an extremely valuable feature for all CDT adopters and plugin developers as it makes extremely easy to extends highlighting in a consistent and performant way. We consider this most "contribution-ready" change but it still needs some refinements.

3. Code formatting was breaking code in our language and crashing with exception. We were reluctant to fork the whole format infrastructure so the change https://github.com/eugeneo/cdt/commit/1da0268557ac85505141e4de4532c7059160173e opens up default formatter for extension. We would also like to suggest creating "language specific" formatters. E.g. then we would contribute our custom processor to the default CDT formatting and our processor will be used behind the scene without the need for the user to select it on the preference page (or us overriding preference in the product).

4. Refactorings did not know how to print our custom AST nodes. I.e. our custom function call node was treated as basic C++ function call and stripped from all important semantics. Variables would loose our extended attributes, etc. https://github.com/eugeneo/cdt/commit/c3a9d6b74a8e69cec7ca42f449a589cfd76fb6e0 allows ASTNodes implement IWritableNode interface so refactorings could use custom writers to output those nodes.

5. Some refactorings (in our case it was Extract Function) should be aware of the custom semantics (i.e. in our case newly created function needs some custom attributes deduced from the original location of the extracted code). https://github.com/eugeneo/cdt/commit/3df6d2d39e229b3f94a8c56bc0a2d9f346260230 allows contributing custom language-specific "refactoring delegates" that are used on the source code written in the specified language. Currently only Extract Function can be extended this way but I would be glad to retrofit others as well.

I would like to know your opinion on what features you think make sense in the upstream project and then I will open a new entries in the BZ where we will discuss implementation details and scope. I've also seen some requests for extended semantic highlights that I could try to implement on top of our extension point (see (2) above).

Best regards,

Eugene