Community
Participate
Working Groups
I20050920 When possible we should provide the AST near the completion location.
We in JDT UI/Text would like to use the context for the following: - better selection of which templates to show: e.g. template for new methods only in type bodies, 'for' template only when completing a statement ect. - get information about the environment (what other fields, methods are in the scope) for better fill in values in templates (argument guessing) and for better getter/setter proposals (know the fields to create getter/setters, avoid duplicates) - allow more context aware proposals (similar to the APT efford). E.g. limit possible argument values for a method, e.g. In SWT: new Button(composite, | : suggest SWT.PUSH, SWT.RADIO ect...) -------------------------------------------------------------------------- This is a proposal how to structure a context element, in case we don't want to provide an AST: 1. what are we completing: a set of the following method declaration field declaration type declaration statement expression type reference method reference constructor reference field reference string literal class A { fo| : type decl | method decl | field decl | type reference int fo| : method decl | field decl { fo| : type decl | method decl | field decl | statement | expression | type ref | method ref | field ref if (fo| : expression | method ref | field ref foo(| : expression | method ref | field ref foo("x| : string literal x.f| : method ref | field ref A<S,| : type ref new X : type ref | constructor ref } } 2. what is the current environment - existing or new construct class A { public int fo| : new public int fo|() {} : existing } - start/end offsets class A { public int fo| : start with [p, end after o (at invocation offset) public int fo|() {} : start with [p, end after } } - parent chain chain of the following elements and properties: type decl: declared methods and types (e.g. signatures, or IBindings?) kind (enum, annot...), is anonymous method decl: method signature (parameters, exceptions) method invocation argument: parent method declaration(s), index (which argument) class A { fo | : type A public void foo() { fo| : method foo() / type A x.bar(| : invocation argument 0 in 'bar' of 'X' / method foo() / type A x.bar(1, getY().o("bl| : invocation argument 0 in 'o' of 'Y' / invocation argument 1 in 'bar' of 'X' / method foo() / type A } - expected type (for expressions)
Locations for annotations: annotation parameter key annotation parameter value class A { @XY(| : expression | type ref | annotation parameter key | annotation parameter value @XY(x=| : expression | type ref | annotation parameter value Parent chain elements for annotations: annotation: annotation type, which argument @interface T1 { String value(); } @interface T2 { String value(); T1 name(); } class A { @T1(| : argument 'value' in T1 / foo1 / A void foo1() { } @T2(value=1, key=@T1(| : argument 'value' in T1 / argument 'key' in T2 / foo1 / A void foo2() { } } Theodora, maybe you want to review if this would cover all the annotations information you require.
martin - the comment 1 contains a proposal of context information required by JDT/UI or is it a more general proposal ?
I tried to make a general propasal that would work for JDT and APT. See the first section of commment 2 for our requirement, so if there's a different way to solve that, this would be fine as well.
(In reply to comment #4) > See the first section of commment 2 for our requirement I believe you meant the first section of comment 1.
In reponse to <a href="#c2">comment #2</a> The context proposal looks pretty close to what we discussed in Zurich and will mostly satisfy needs from APT. I have two questions though. 1) Do we get a "completion token"? For example: @XY( x = "bl| The "completion token" would be the String "bl", which is the token to the lexer before the invocation site. This information may be be interesting to completion computers. If this information is included in the context as a token, for example, computers will exclude anything that doesn't start with "bl". Another example would be: @XY( fo| or even @XY( abc.fo| where fo could be the prefix of a type or a constant field etc. The next thing this would lead to is binding information for the "completion token". Is this too much of a Java problem and not general completion context anymore? 2) starting and ending offset. Example : @XY( fo| What would be the starting and ending offset in this case? I would expect it to be the same as the "completion token". Your example in <a href="#c1">comment #1</a> seems to suggest that starting offset would be that "@" and end at invocation offset. What about for @XY( fo| )? Would the ending offset be after the ")"? I would expect a completion computer would use the starting and ending offset to calculate the replacement offset in the completion proposal. If that's the case, when the offsets are wider than the "completion token", there seems to require a way for the completeion computer to format code around the "completion token", which sounds complicated and problematic.
1) We will give the 'completion token' with the java completion context. With your test case we will give "bl". We will give information about the qualifier or receiver type of 'completion token' but currently we didn't decide the way to give this information (binding, unique key, type signature, java element, ...). I am not sure to understand the second question. This bug is use to discuss about the java completion context (org.eclipse.jdt.core.CompletionContext). There is another bug in JDT/Text to dicsuss about more general completion context (bug 110913). 2) The completion context will give at least the 'completion token' starting and ending offset. Example: foo(tok|en) the completion token is 'tok' the completion offset is at 'e' location the completion token start is at 't' location the completion token end is at 'n' location
In reply to comment 6 and comment 7: Note that you can also ask the UI completion context to provide the java identifier preceding the completion offset.
David- thanks for the clarification on the "completion token". The information is exactly what I am looking for, both the "token" as well as the offsets. In comment #6, the second question refers to item 2 in comment #1 from Martin. In particular, the section that says the following. "- start/end offsets class A { public int fo| : start with [p, end after o (at invocation offset) public int fo|() {} : start with [p, end after } } " This section suggests that the start and end offset in the completion context are wider than the range of the "completion token". Are we talking about two separate sets of offsets? Martin?
I forgot the normal completion token offsets with should always be there, agree. The range I descibed would give some extra offsets for cases like method/field/type declarations where the token offsets are not enough. e.g. public static int fo| or class A| Here we would like to introduce a new method/type including a comment and have to replace starting from 'public'. Having just the token 'fo' would make it hard to find out where the (possible) method declaration really starts.
Completion context can give information about completion token. Proposed API: public class CompletionContext { ... /** * The completion token is a name. */ public static final int TOKEN_KIND_NAME = 1; /** * The completion token is a string literal. */ public static final int TOKEN_KIND_STRING_LITERAL = 2; /** * Returns the completed token. * This token is either the identifier or Java language keyword * or the string literal under, immediately preceding, * the original request offset. If the original request offset * is not within or immediately after an identifier or keyword or * a string literal then the returned value is <code>null</code>. * * @return completed token or <code>null</code> */ public char[] getToken(){} /** * Returns the kind of completion token being proposed. * <p> * The set of different kinds of completion token is * expected to change over time. It is strongly recommended * that clients do <b>not</b> assume that the kind is one of the * ones they know about, and code defensively for the * possibility of unexpected future growth. * </p> * * @return the kind; one of the kind constants declared on * this class whose name starts with <code>TOKEN_KIND</code>, * or possibly a kind unknown to the caller */ public int getTokenKind(){} /** * Returns the character index of the start of the * subrange in the source file buffer containing the * relevant token being completed. This * token is either the identifier or Java language keyword * under, or immediately preceding, the original request * offset. If the original request offset is not within * or immediately after an identifier or keyword, then the * position returned is original request offset and the * token range is empty. * * @return character index of token start position (inclusive) */ public int getTokenStart(){} /** * Returns the character index of the end (exclusive) of the subrange * in the source file buffer containing the * relevant token. When there is no relevant token, the * range is empty * (<code>getTokenEnd() == getTokenStart()</code>). * * @return character index of token end position (exclusive) */ public int getTokenEnd(){} ... }
Completion context can give information about completion location Proposed API: public class CompletionContext { ... /** * The completion is requested inside a Javadoc and is not in a more specific location. */ public static final int INSIDE_JAVADOC = 1; /** * The completion isr equested inside a type declaration header and is not in a more specific location. * The completion offset is between the type name and the opening brace of the body */ public static final int INSIDE_TYPE_DECLARATION = 2; /** * The completion is requested inside a type body and is not in a more specific location. * The completion offset is between the opening brace of the body and the ending brace */ public static final int INSIDE_TYPE_BODY = 3; /** * The completion is requested inside a method declaration header and is not in a more specific location. * The completion offset is between the opening parenthesis and the opening brace of the body */ public static final int INSIDE_METHOD_DECLARATION = 4; /** * The completion is requested inside a method body and is not in a more specific location. * The completion offset is between the opening brace of the body and the ending brace */ public static final int INSIDE_METHOD_BODY = 5; /** * The completion is requested inside an annotation and is not in a more specific location. */ public static final int INSIDE_ANNOTATION = 6; /** * The completion is requested inside a field initializer and is not in a more specific location. */ public static final int INSIDE_FIELD_INITIALIZER = 7; /** * The completion is requested inside an intializer and is not in a more specific location. */ public static final int INSIDE_INITIALIZER = 8; /** * The completion is requested inside a compilation unit and is not in a more specific location. */ public static final int INSIDE_COMPLATION_UNIT = 9; /** * Returns the offset position in the source file buffer * after which code assist is requested. * * @return offset position in the source file buffer */ public int getOffset(){} /** * Returns the kind of completion location. * <p> * The set of different kinds of completion location is * expected to change over time. It is strongly recommended * that clients do <b>not</b> assume that the kind is one of the * ones they know about, and code defensively for the * possibility of unexpected future growth. * </p> * * @return the kind; one of the kind constants declared on * this class whose name starts with <code>INSIDE</code>, * or possibly a kind unknown to the caller */ public int getLocationKind(){} ... } Notes: -The getLocationKind() give only the general kind of the completion area. We must add some API to describe the completion node area. -getLocationKind() could return a bit mask instead of constants.
Completion context can give information about the fields, methods and locals variables which are in the scope Proposed API: public class CompletionContext { ... /** * Return names of fields in the scope. */ public char[][] getFieldsName() {} /** * Return types of fields in the scope. * Dimension of the returned array is the same as the array returned by {@link #getFieldsName()} */ public char[][] getFieldsType() {} /** * Return declaring types of fields in the scope. * Dimension of the returned array is the same as the array returned by {@link #getFieldsName()} */ public char[][] getFieldsDeclaringType() {} /** * Return names of local variables in the scope. */ public char[][] getLocalsName() {} /** * Return types of local variables in the scope. * Dimension of the returned array is the same as the array returned by {@link #getLocalsName()} */ public char[][] getLocalsType() {} /** * Return names of methods in the scope. */ public char[][] getMethodsName() {} /** * Return signatures of methods in the scope. * Dimension of the returned array is the same as the array returned by {@link #getMethodsName()} */ public char[][] getMethodsSignature() {} /** * Return declaring types of methods in the scope. * Dimension of the returned array is the same as the array returned by {@link #getMethodsName()} */ public char[][] getMethodsDeclaringType() {} ... }
I'm not sure if I fully understand the proposal. Can you clarify what conclusions I can makes when I'm at a certain location (getLocationKind(), INSIDE_xx) e.g. like INSIDE_METHOD_DECLARATION? Am I completing the method name, or the return type, or is it a parameter type, a parameter name, or a parameter modifier? or for INSIDE_METHOD_BODY: Does that mean that I can enter a statement? Or would I still enter statements at places where only expressions should be? (e.g. if (|): problem is to not suggest a for-code template.
getLocationKind() give you the area of the completion offset. That doesn't describe the 'completion node'. If the locationKind is INSIDE_METHOD_DECLARATION then the offset if between an open parenthesis of a method declaration and the open brace of this method body (or the end of the method if there is no method body). The completion offset can be inside a parameters type, parameters name, exceptions type). If the offset is before the open parenthesis, this is INSIDE_TYPE_BODY. That's because the code before the completion offset can be seen as the begin of a method or a field ('int foo|(int i)' can be seen as 'int foo|'. INSIDE_METHOD_BODY only signals that the completion offset is inside a method body. It can be a statement or an expression or something else. /*start of INSIDE_COMPILATION_UNIT*/ public class X/*end of INSIDE_COMPILATION_UNIT*//*start of INSIDE_TYPE_DECLARATION*/ extends Y /*end of INSIDE_TYPE_DECLARATION*/{ /*start of INSIDE_TYPE_BODY*/ Object var /*end of INSIDE_TYPE_BODY*/= /*start of INSIDE_FIELD_INITIALIZER*/new Object();/*end of INSIDE_FIELD_INITIALIZER*/ /*start of INSIDE_TYPE_BODY*/ int foo/*end of INSIDE_TYPE_BODY*/(/*start of INSIDE_METHOD_DECLARATION*/int a, int b) throws E /*end of INSIDE_METHOD_DECLARATION*/{ /*start of INSIDE_METHOD_BODY*/ foo(1, 2); return 0; /*end of INSIDE_METHOD_BODY*/ } | This API only give you a general information. I will add some others API to give more specific information. And this API will be significant only in specific location kind. eg. I could add 'CompletionContext#isExpession()' and this API could be only valid if location kind is INSIDE_METHOD_BODY. eg. i could add 'CompletionContext#getAST()' only if location kind is INSIDE_ANNOTATION.
Respond to Comment #15 If you add CompletionContext#getAST(), would we get back to the AST definition problem that we tried to avoid in the first place? Also, will the AST have binding information? When CompletionContext#getTokenKind() returns a TOKEN_KIND_NAME, the next question one would ask is, what exactly does the name refer to? Is it a field reference, type or package etc? Some binding information, IBinding, signature of some sort would be nice.
Respond to Comment 16 Currently i'm not sure if we can return an AST. But if we return an AST it will be only the annotation's AST and not the full AST. Annotation's AST seems more simple. I need to do more investigation on this front because currently our internal completion AST contains the completed node but doesn't contains enough context information (enclosing annotations, sibling annotations, annoted node). We won't return a binding (or signature or ...) for the completion token itself because this token is incomplete by nature. But we will give a binding (or signature or ...) for the qualifier (or receiver) of this token. eg. var.toStr| we cannot return a binding for 'toStr' but we can for 'var'.
Some more questions: Would the type informations be returned in signature or key notation (seems the right thing to do). So far I don't really see the benefit of getLocationKind(). It seems to me that the returned information leaves too many possibilities and it is not clear what conclusions a participant can make from that result. I still think the interesting information is: - what kind of completions make sense at this location (comment 1, 1) - what is our context _chain_ regarding to: - inside which method invocation - inside which annotation - inside whicht method - inside which type
It seems a bit odd that you will provide ast for annotation completions but not for other cases. Also, would this also be redundant information given that the parent chain elements for annotations will be available? see Comment #2. What additional information would the AST give us that's not in the current proposed completion context. Am i missing something?
Theodora, in case this was't clear: my comments (comment 1 and comment 2) represent the jdt.ui/text suggestions, with the idea to convince jdt.core (david) to design the API in a way that is as powerful. We haven't had a feedback on those suggestions and wonder in what directions jdt.core wants to go. I also hope that we can agree on a nicely designed API not just solves the annotation case (e.g. with an AST just for that) but that is as useful for other completion participants.
I released the part described in comment 11 (information about completion token) Tests added CompletionContextTests CompletionContextTests_1_5 JavadocCompletionContextTests JavadocCompletionContextTests_1_5 Tests updated CompletionParserTestKeyword
Respond to Comment 20 About the AST. I tried to give an AST only for annotation because it is probably possible to build it for an annotation. But build an AST for all contexts is a too strong requirement. If it isn't a good idea to give an AST only when we can then we won't give any AST. Respond to Comment 18 The first purpose of getLocationKind() is to be able to specify what others context API will be valid in the current context. If finally we haven't any location specific API, i'm agree that this method isn't very useful. Respond to Comment 20 About the proposal of comment 1 and comment 2. Your proposal is very intersting and i use it as base for my investigation. Propose a parent chain is the way where i want to go. My plan is to propose only the parent of the completion. In a second step i should be able propose also the enclosing declaring element (method, field, variable or type declaration) In a last step i would try to propose also the complete parent chain but this require more work because currently our internal mechanism doesn't keep these data. Another part of your proposal is to propose the kind of completions make sense at this location. I am agree that we need to give more information about the 'completion node' itself but i don't think that the purpose of the completion context is to decide what completion make sense. I think that the completion context must give information to let the participant to decide what completion make sense.
It's too late to add new API in 3.2. Defer post 3.2.
At the JDT Summit 06 we discussed the following approaches: - Provide IJavaElements with updated source positions (problem: Will have to use a special working copy owner) - Provide a sparse AST DOM (only declarations, no statements) - Provide a statement recovered AST DOM with nodes for the current method (we will have to find out if statement recovery can really help here) - Provide a custom AST DOM (new AST level) with extra recovery nodes
Is there any way to get context information *today* in eclipse 3.2 ?
Things I would like to access from custom code completion is as follows: In general: Support methods for getting the complete string value and an relative offset. e.g. x.someMethod("here " + "is a string" + "with no! code") Assuming I codecompleted at the ! then I would like to get "here is a string with no! code" and the number 25. In annotations: @NamedQueries( @NamedQuery(query="fr<code-complete> What annotation am I currently in (@javax.persistence.NamedQueries/@javax.persistence.NamedQuery) What attribute am I currently on (query) @Table(name="<code-complete> What annotation am I currently in (@javax.persistence.Table) What attribute am I currently on (query) In code: void someMethod() { em.createQuery("fr<code-complete> What method "context" am i currently *on* (EntityManager.createQuery) What parameter is it for (parameter 1, of string) What method am I currently *in* (someMethod) I know that alot of this info is available to the builtin code completion so just getting hints about where to get access to this for 3.2 would be very much appreciated.
Would it be possible to get information needed for bug 49552 (see bug 495522 comment 3).
Topic discussed and accepted in JDT/Summit
Here is a summarize of the current state of this bug. Currently the CompletionContext already give the following information - Completion token kind: name or string literal - Completion token source location - Completion general location: Javadoc or code - Expected types: If you do code assist in right hand side of an assignment expected type is type of the left hand side. Information that are identified as useful but currently not given by the completion context - Enclosing context of the completion: * Inside which method declaration the completion occurs * Inside which type declaration the completion occurs - Annotated element for an annotation - Sibling elements: * Sibling methods/fields/types * Sibling annotations - Information about the completion node itself * The nature of the completion node * The invocation chain (eg: foo(bar(var|)) * The annotation chain (eg @Annot1(@Annot2(@Annot3|)) Current limitation or issues - We can’t return a full DOM AST as context because: * The internal completion AST is incomplete * The internal completion AST contains fake nodes for completion purpose * We don’t want to decrease performance to create a full and correct AST. - The annotation recovery is weak. It must be improve - For statement completion, only nodes near completion point are rebuilt: the completed nodes and sometimes his parent node For ‘foo(bar(var|))’ only nodes for ’bar(var|)’ are rebuilt We want to implement this feature by giving context information with several specific API instead of a single full DOM AST. Do you see some other missing context information for JDT Text and APT needs ? And what information do you absolutely need ?
Re: specific API vs. full DOM AST (which we do not have) We mentionned the possibility of providing DOM AST bindings, for the enclosing method. This would allow to find out about parents, and siblings. This wouldn't reveal local variables, but these could be surfaced through a different API (i.e. a list of local variable bindings in scope, likely including outer local variables from enclosing types as well).
All looks good. I assume this is should all be possible in 3.2 also (but of course not as direct api). Any hints on how to get similar info in 3.2. would be appreicated.
There is no plan for 3.2, as no API addition may occur in a maintenance stream. As to reaching internals for equivalent functionality in 3.2, I think you'd have to reach deep inside, since the context has a different life cycle than the codeassist operation which had the information you're looking for (i.e. by the time you get your hands on the context, the data is gone).
...but if I know the compilationunit and the position I am in I should be able to build/access the information, right ?
Our completion engine use principally a resolved AST to compute his information A specialized parser is used to build this AST. This AST is slightly different of the DOM AST you can build with the API ASTParser, especially near the completion position where the nodes are very different. And a lot of internal methods of the compiler are also used. So i don't think you will be able to easily compute the same information from outside JDTCore. Most of the information are build inside CompletionEngine#complete(ICompilationUnit sourceUnit, int completionPosition, int pos). You can look this method to understand what is done to build this information.
will look there. thanks.
The new APIs of bug 202467 and bug 202470 solved our problems regarding more context for code assist. Ok to close for me.
(In reply to comment #36) > The new APIs of bug 202467 and bug 202470 solved our problems regarding more > context for code assist. Ok to close for me. Dani, it looks like bug 49552 is still dependent on this bug. So can this bug really be closed?
I suggest to close this. Bug 49552 is P5 and we can file a new bug report requesting more context once we decide to attack that bug.
Fixes for bug 202467 and bug 202470 add the following API CompletionContext#getTokenLocation() CompletionContext#getEnclosingElement() CompletionContext#getVisibleElements(String) These API solve most of the problems described in this bug report then we will consider this bug as fixed. If you need some other new API then file a bug report for each requested API. I close this bug as WORKSFORME because bug 202467 and bug 202470 have been fixed in 3.4.
Verified for 3.5M1 using I20080805-1307
