Re: [jdt-core-dev] start position in DOM

[Martin Aeschlimann <martin_aeschlimann@xxxxxxxxxx>]

For 3.0 M1/M2 I implemented a heuristic
how to assign comments to a node for the AST rewriter. The positions are
evaluated on an existing AST by re-scanning. Of course it would be much
more efficient to do this when creating the AST so I would be very interested
if we can share the implementation. The implementation is in TokenScanner.getTokenCommentEnd
and TokenScanner.getTokenCommentStart.

The rules are:
Comments ahead of a node are assigned
to the node
- as long they are not separated by
empty lines (example 1) and the comment does not start on the same line
as the end position of the previous token (example 2) except if this is
the same line as the node starts (example 3)
- exception is the first javadoc comment
directly ahead of declaration nodes: It can be separated by new lines (example
4)

 Comments after a node are assigned
to the node if
 - they are on the same line as
the node end position (example 2) but the next node does not start on this
line (example 3)
 - all comment following directly
on next lines with no empty line in between (example 5), as long as they
are followed by a empty line else they don't belong to the next node (example
2)

Examples:
-------- (example 1)
NODE 0;

// comment (not assigned to a node,
empty line are separating)

// node1 (comment directly ahead of
comment of node1, no separation by empty lines)
// node1 (comment directly ahead of
node1, no empty lines)
NODE1; 

--------   (example 2)

NODE0; // node 0  (end comment
on same line as node0)
//node1 (directly ahead of comment of
node1: binds stronger than being directly after comment of node 0)
//node1 (comment directly ahead of node1,
no empty lines)
NODE1;

------  (example 3)

NODE0; /* node 1 */ NODE1; (start comments
bind stronger that end comments)

------ (example 4)

// node 0 (directly following the next
comment, no line between)
/** Javadoc comment binds to next node
even if there are empty lines after
*/

NODE0;

-------- (example 5)

NODE1; // node 1 (starting on same line
as end position of node)
// node 1 (end comment on same line
as node)
// node 1 (end comment directly following
node1 (no empty lines) but separated by a empty line from the next node's
starting comment)

//node2 (comment directly ahead of node1,
no empty lines)
NODE2;

Martin

Philippe P Mulet/France/IBM@IBMFR
Sent by: jdt-core-dev-admin@xxxxxxxxxxx

03.07.2003 10:35

Please respond to
jdt-core-dev@xxxxxxxxxxx

To	jdt-core-dev@xxxxxxxxxxx
cc
Subject	Re: [jdt-core-dev] start position in DOM

Ok, I checked the Type/MethodDeclaration nodes, and their source range
actually includes the leading javadoc (exactly one if any), as I had
guessed from looking at the implementation.
Since it is spec'ed as such, I will leave it as it is. We may want to
provide an extendedStartPosition() which would allow some degree of
consistency with other source ranges in our other APIs.




|---------+------------------------------>
|         |           Jim
des            |
|         |           Rivieres/Ottawa/IBM|
|         |           @IBMCA
            |
|         |           Sent
by:           |
|         |           jdt-core-dev-admin@|
|         |           eclipse.org
       |
|         |            
                 |
|         |            
                 |
|         |           07/02/2003
06:53 PM|
|         |           Please
respond to  |
|         |           jdt-core-dev
      |
|         |            
                 |
|---------+------------------------------>
  >------------------------------------------------------------------------------------------------------------------------|
  |                  
                     
                     
                     
                     
             |
  |       To:       jdt-core-dev@xxxxxxxxxxx
                     
                     
                     
            |
  |       cc:            
                     
                     
                     
                     
         |
  |       Subject:  Re: [jdt-core-dev] start position
in DOM                    
                     
                    |
  |                  
                     
                     
                     
                     
             |
  >------------------------------------------------------------------------------------------------------------------------|




> Is there anything better than this ? BTW, I understand the DOM clients,
and
> will not change their assumption.

DOM AST source ranges are spelled out in AST.parseCompilationUnit
(responsible for setting source ranges):

The returned compilation unit node is the root node of a new AST. Each
node in the subtree carries source range(s) information relating back to
positions in the given source string (the given source string itself is
not remembered with the AST). The source range usually begins at the first
character of the first token corresponding to the node; leading whitespace
and comments are not included. The source range usually extends through
the
last character of
the last token corresponding to the node; trailing whitespace and comments
are not included. There are a handful of exceptions (including compilation
units
and the various body declarations); the specification for these node type
spells out the details. Source ranges nest properly: the source range for
a child is always within the source range of its parent, and the source
ranges of sibling nodes never overlap.





Philippe P Mulet/France/IBM@IBMFR
Sent by: jdt-core-dev-admin@xxxxxxxxxxx
07/02/2003 12:30 PM
Please respond to jdt-core-dev


        To:     jdt-core-dev@xxxxxxxxxxx
        cc:
        Subject:        Re: [jdt-core-dev]
start position in DOM




Is there anything better than this ? BTW, I understand the DOM clients,
and
will not change their assumption.

      /**
       * Returns the character index into the original source
file
indicating
       * where the source fragment corresponding to this
node begins.
       *
       * @return the 0-based character index, or <code>-1</code>
       *    if no source position information
is recorded for this node
       * @see #getLength
       */
      public int getStartPosition()




|---------+------------------------------>
|         |           Jim
des            |
|         |           Rivieres/Ottawa/IBM|
|         |           @IBMCA
            |
|         |           Sent
by:           |
|         |           jdt-core-dev-admin@|
|         |           eclipse.org
       |
|         |            
                 |
|         |            
                 |
|         |           07/02/2003
06:05 PM|
|         |           Please
respond to  |
|         |           jdt-core-dev
      |
|         |            
                 |
|---------+------------------------------>

>
------------------------------------------------------------------------------------------------------------------------|

  |                  
                     
        |
  |       To:       jdt-core-dev@xxxxxxxxxxx
                   |
  |       cc:            
                     
              |
  |       Subject:  Re: [jdt-core-dev] start position
in DOM             |
  |                  
                     
        |

>
------------------------------------------------------------------------------------------------------------------------|





>  Any comments ? I checked the spec on the DOM AST, and it is
only
loosely
> spec'ed, so it could be evolved.

Actually, the DOM AST specs source ranges quite tightly (refactoring needs
exact ranges to do source code rewriting).







Philippe P Mulet/France/IBM@IBMFR
Sent by: jdt-core-dev-admin@xxxxxxxxxxx
07/01/2003 11:11 AM
Please respond to jdt-core-dev


        To:     jdt-core-dev@xxxxxxxxxxx
        cc:
        Subject:        [jdt-core-dev]
start position in DOM



Something annoying me in our current parser implementations is that there
is no consistency in our starting positions amongst our different type
of
nodes (compiler ast, dom ast, source elements, jdom).

This comes from the fact there are 2 implementations of
Parser#checkAnnotation which are subtily different. The JavaModel is
considering all comments behind previous element to be part of the next
element (taking into account trailing line comment). I would like to
generalize this behavior to others as well. The DOM only contains the
immediate leading javadoc comment, and we could enclose them all as we
do
in the model.

Any comments ? I checked the spec on the DOM AST, and it is only  loosely
spec'ed, so it could be evolved.


_______________________________________________
jdt-core-dev mailing list
jdt-core-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/jdt-core-dev



_______________________________________________
jdt-core-dev mailing list
jdt-core-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/jdt-core-dev





_______________________________________________
jdt-core-dev mailing list
jdt-core-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/jdt-core-dev



_______________________________________________
jdt-core-dev mailing list
jdt-core-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/jdt-core-dev





_______________________________________________
jdt-core-dev mailing list
jdt-core-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/jdt-core-dev