Community
Participate
Working Groups
PROBLEM A whitespace-tokenizing PublicScanner returns a whitespace token when the whitespace starts one position after the end position specified by resetTo(). It should return EOF. Below is some test code that tickles the bug, along with its output. I ran it inside a HelloWorld plug-in. I also give a workaround for clients and a possible, untested fix. CODE public void testScanner() throws InvalidInputException { IScanner s = ToolFactory.createScanner (true, true, true, "1.5", "1.5"); System.err.println("IScanner is " + s.getClass()); char[] source = {';', ' '}; s.setSource (source); s.resetTo (0, 0); s.getNextToken (); System.err.println("token is ;? " + ";".equals (new String (s.getRawTokenSource ()))); int t = s.getNextToken (); System.err.println("token is EOF? " + (ITerminalSymbols.TokenNameEOF == t)); } OUTPUT IScanner is class org.eclipse.jdt.internal.core.util.PublicScanner token is ;? true token is EOF? false WORKAROUND Clients need to check for EOF and tokens whose start position is after end. For example: int token; IScanner scanner = ToolFactory.createScanner (true, true, true, "1.5", "1.5"); scanner.setSource (source); scanner.resetTo (start, end); token = scanner.getNextToken (); while (token != ITerminalSymbols.TokenNameEOF && scanner.getCurrentTokenStartPosition () <= end)) { // ... token = scanner.getNextToken (); } POSSIBLE, UNTRIED FIX In PublicScanner.getNextToken(), just bofore consuming whitespace, check for EOF. ... int whiteStart = 0; try { // FIX? if (this.currentPosition >= this.eofPosition) { return TokenNameEOF; } // END FIX? while (true) { //loop for jumping over comments this.withoutUnicodePtr = 0; ...
This could be a candidate for 3.1.1
If not causing harm to any client tests (UI, refactoring, etc...) then +1 for 3.1.1.
Created attachment 25922 [details] Proposed fix
This fix passed all existing tests including jdt/ui and refactoring.
+1 for 3.1.1
Remember to update both the master and public scanners.
Fixed and released in 3.1 maintenance stream. Regression tests added in org.eclipse.jdt.core.tests.compiler.regression.ScannerTest.test036/test041. Will be released in HEAD post 3.2M1.
You guys are great!
Fixed and released in HEAD. Same regression tests as 3.1.1.
Verified in I20050920-0010 for 3.2M2
Verified using M20050923-1430 for 3.1.1