|Re: [cdt-dev] Suggestions for dealing with tests|
Hm... Actually, I got it differently, regarding handling of numbers.
The grammar for pp-number (preprocessor numbers) is as follows [lex.ppnumber]:
pp-number e sign
pp-number E sign
Therefore, everything starting with a digit or a dot and a digit is a number. Thus .55.4h5ze+E-5gg would be a pp-number, even though it will not be convertible into a meaningful floating number.
I guess the way Markus suggested was to rip the part districting the kinds of numbers out of the preprocessor, leaving it yielding pp-numbers and move it to the parser.
Regarding the tests: At the moment I have 17 tests that error in a suite that tests included files (I forget which it is exactly), and a few that fail because there aren't any tests in the suite (base classes I think). I'm assuming the latter is because I'm running the suite incorrectly: I'm right-clicking on the package and Run As->jUnit Plugin Test.
For the UDL implementation, I understand what you're saying about not introducing new elements, which is what my first attempt tried to do and I had problems with, but now I look at it again, I think I understand how you're suggesting to implement it.
So, if I understand correctly, the lexer should return two tokens for numbers with suffixes: the number, and an identifier. And it would be the job of the Parser to combine these into one ASTNode. I agree, that does seem like the best approach, and would mean that the places (I've counted at least two, Lexer.java and CPreprocessor.java) that have more-or-less the same code for parsing numbers can be in just in the place they belong in the lexer, and the AST token can be used in CPreprocessor to determine if the token is a UDL, by adding and extra method, isUserDefined, and maybe other methods such as getNumberType, which would return INT, HEX, BINARY, etc. The other method I see that could be useful is isMalformed() for use in CPreprocessor.
I think the code for all this is already written between Corbat and myself, and just needs placing in the right places and testing.
I had a closer look at the standard and I agree with you:
pp-number contains any number including user defined literals. But for preprocessing the characters and strings are distinct from user defined characters and strings, considering preprocessing tokens (from translation phase 3).
Looking at translation phase 7 (conversion of preprocessing tokens to tokens) user defined literals are distinct from the other kinds of literals [lex.literal.kinds], with the subcategories int, float, char and string.
From the description the conversion of the tokens seems to me like a pre-step of phase 7. But I guess it could be done on the fly, while parsing as well.
A user-defined literal is still of kind integer, char, floating-point or string. So itâd be natural to add the property isUserDefined() to IASTLiteralExpression.
Yes, dealing with the suffix is the parsers task. By the standard itâs even the task of the parser to distinguish between the various kinds of number literals. I donât exactly know why this is done in the preprocessor. Itâd be actually nice to move this task away from it, which would collapse the various kinds of number tokens to one preprocessor number token as described in 2.10.
However, the LR-Parsers probably rely on receiving the tokens as they are delivered today. So we could have an option whether the preprocessor shall classify the number tokens or not. For the GNUCPPSourceParser we could then move the classification into the parser.
Exactly, CPPASTLiteralExpresion.getExpressionType() needs to be changed. Plus, as mentioned before, by using IASTImplicitNameOwner the literal _expression_ can provide the binding to the operator that is called to create the object denoted by the literal.
Regarding the UDL implementation:
So your suggestion, Markus, is to lex user defined literals as tSTRING, tCHAR, tINTEGER and tFLOATINGPT?
Shall IASTExpression get a new kind for user defined literals?
Recognizing the suffix will be task of the parser then, right?
I guess getExpressionType() in CPPASTLiteralExpression must be extended to return the type of the resolved literal operator.
That probably makes the implementation a bit easier.
The preprocessor is not run in a specific language. In case there needs to be different behavior for C and C++ you need to introduce an option which should be controlled via IScannerExtensionConfiguration (The GPPLanguage and the GCCLanguage provide different objects for this configuration). The testcase needs to be elaborated to test with the two different configurations.
For the case, where we allow user defined literals the checkNumber method needs to behave differently. I also think that the classification of the number literals in the lexer needs to be changed, however it can be done, such that it works independently of whether user-defined literals are allowed or not.
My recommendation is to neither introduce new kinds of tokens nor to create a new IASTNode. I think it is sufficient to let CPPASTLiteralExpression implement IASTImplicitNameOwner, which allows to provide the references to the implicit function calls.
The tests nested in org.eclipse.cdt.core.suite.AutomatedIntegrationSuite should all pass.
On Behalf Of Richard
I've been working on getting user-defined literals working syntactically for the past week, I believe I'm in the home stretch as I'm now working through the failing test cases under org.eclipse.cdt.core.tests.
I have a few questions regarding this.
- Firstly, how do I deal with tests that are supposed to fail in C mode, but pass in C++ mode?
The current test that demonstrates this is PerprocessorTests.testGCC43BinaryNumbers.
There are 5 binary literals that are tested for failing: 0b012, 0b01b, 0b1111e01, 0b1111p10, 0b10010.10010
With UDLs the first and the last of these should fail, and the middle three can be considered binary literals with UDL suffixes. But in C mode, or C++ sans UDL, they all should fail. Is there a way to test which language the current test is being run in?
- Lastly, I've been testing my branch against master, and I've noticed there are a fair number of tests with errors or failing. Is this expected, or do I have my project set up incorrectly?