Bug 106403

Description Stoney 2005-08-08 16:10:54 EDT

PROBLEM

A whitespace-tokenizing PublicScanner returns a whitespace token when the
whitespace starts one position after the end position specified by resetTo(). It
should return EOF. Below is some test code that tickles the bug, along with its
output. I ran it inside a HelloWorld plug-in. I also give a workaround for
clients and a possible, untested fix.

CODE

public void testScanner() throws InvalidInputException
{
   IScanner s = ToolFactory.createScanner (true, true, true, "1.5", "1.5");
   System.err.println("IScanner is " + s.getClass());
   char[] source = {';', ' '};
   s.setSource (source);
   s.resetTo (0, 0);
   s.getNextToken ();
   System.err.println("token is ;? "
      + ";".equals (new String (s.getRawTokenSource ())));
   int t = s.getNextToken ();
   System.err.println("token is EOF? "
      + (ITerminalSymbols.TokenNameEOF == t));
}

OUTPUT

IScanner is class org.eclipse.jdt.internal.core.util.PublicScanner
token is ;? true
token is EOF? false

WORKAROUND

Clients need to check for EOF and tokens whose start position is after end. For
example:

   int token;
   IScanner scanner =
      ToolFactory.createScanner (true, true, true, "1.5", "1.5");
   scanner.setSource (source);
   scanner.resetTo (start, end);
   token = scanner.getNextToken ();
   while (token != ITerminalSymbols.TokenNameEOF
          && scanner.getCurrentTokenStartPosition () <= end))
   {
      // ...
      token = scanner.getNextToken ();
   }

POSSIBLE, UNTRIED FIX

In PublicScanner.getNextToken(), just bofore consuming whitespace, check for EOF.

   ...
   int whiteStart = 0;
   try {
      // FIX?
      if (this.currentPosition >= this.eofPosition) {
         return TokenNameEOF;
      }
      // END FIX?
      while (true) { //loop for jumping over comments
         this.withoutUnicodePtr = 0;
         ...