Summary: | Scanner does not report whitespace tokens at end of input | ||||||
---|---|---|---|---|---|---|---|
Product: | [Eclipse Project] JDT | Reporter: | Andre Weinand <andre_weinand> | ||||
Component: | Core | Assignee: | Olivier Thomann <Olivier_Thomann> | ||||
Status: | VERIFIED FIXED | QA Contact: | |||||
Severity: | normal | ||||||
Priority: | P3 | ||||||
Version: | 2.0 | ||||||
Target Milestone: | 2.1 M3 | ||||||
Hardware: | PC | ||||||
OS: | Windows 2000 | ||||||
Whiteboard: | |||||||
Attachments: |
|
Description
Andre Weinand
2002-04-16 12:07:24 EDT
If you have an input like: "package com.ibm.itp.compare.ui; //foo \r\n" I would say the the right tokens at the end are a line comment token followed by EOF. \r\n should be part of the line comment. If this is not the case, we have a bug, but the bug is not that we don't report whitespaces. for the original input "package com.ibm.itp.compare.ui; //foo \r\n" the length of the LINECOMMENT token is 6, that is the space is included but the CrLf is not. for this input "package com.ibm.itp.compare.ui; " the last two tokens reported by getNextToken() are a SEMICOLON (with length 1) and EOF (BTW with length 1 too). So whitespace at the end of input is not reported. Do you expect the line separator to be reported as part of the line comment or as a white space? I assume that lines separators are not reported for multi line comments. Consequently I would expect similar behavior for single line comments: as part of the line comment. So for a source like: "package com.ibm.itp.compare.ui; // toto \r\n" You expect: PACKAGE WS Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 SEMICOLON WS COMMENT // no WS between COMMENT and EOF EOF And with: "package com.ibm.itp.compare.ui; \r\n": PACKAGE WS Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 Token: 6 Token: 5 SEMICOLON WS // this WS includes the line separator. EOF We have a side-effect in the code formatter when the line separator is part of the line comment. I need to investigate more on this side before I can release anything. Defer Created attachment 2086 [details]
patch for the scanner
Can fix it and patch the formatter to get rid of the side-effect. Fixed and released in 2.1 stream. Regression tests added. Verified. |