Community
Participate
Working Groups
We observe a regression from the fix for bug 560392 in our product. In particular, we cannot evaluate the expression "return 7 + 5;" in xtext generated Java code, during one of our automatic regression tests. We observe a variable "arg$1" in the lambda scope, from more or less code as follows: public void execute(final Interface it) { // LINE INFO 16 final Runnable _function_1 = () -> { // LINE INFO 17 it.println("Yeah!"); // LINE INFO 18 it.add(1); // the evaluation is done at a breakpoint here }; // LINE INFO 19 final Runnable _function_2 = () -> { // LINE INFO 20 it.println("OMG!"); // LINE INFO 21 it.add(2); }; } The variable name "1" is computed from "arg$1" during the new code in ASTEvaluationEngine.getCompiledExpression(String, IJavaStackFrame), which leads to a compile problem in the compilation unit: Pb(235) Syntax error on token "1", invalid VariableDeclaratorId From that point on the evaluation snippet doesn't work (I assume any snippet won't evaluate). So far we have no reproduction snippet unfortunately. When I use the snippet from above in a minimal Java project, the variable is called "val$it" and the fix for bug 560392 works as intended. We are looking into the problem.
Sarika, any idea why we would have a variable named "arg$1" and not "val$it" while in the lambda for "_functional_1" (see the snippet in the description)?
I see that in JDIStackFrame.LambdaASTVisitor.visit(LambdaExpression), the following check prevents the update of the lambda variable name to the name expected by the fix in bug 560392: // check if the lineNo fall in lambda region, it can either be single or multiline lambda body. if (lineNo < cu.getLineNumber(lambdaExpression.getStartPosition()) || lineNo > cu.getLineNumber(lambdaExpression.getStartPosition() + lambdaExpression.getLength())) { return true; } lineNo is equal to the line in the xtext DSL file, but the compilation unit is the source file generated by xtext, where the line numbers are different. So the if expression evaluates to true, and the renaming of the lambda variables is skipped. This breaks the assumptions in the new code from bug 560392.
Christian, any suggestions here? IMO this is one of the following: * JDIStackFrame should report a line number from the generated source. * The compilation unit should report lines that match the xtext DSL. Though no idea what might break due to either of those. Or is some adapter call missing in JDIStackFrame.LambdaASTVisitor.visit(LambdaExpression), that makes the xtext magic work as expected?
@Simeon - Can you add example snippet of generated java source code which cause this issue ? - Also can you reproduce this issue with pure java project ? - Does your xtext project use maven for dependency management ?
> - Can you add example snippet of generated java source code which cause this > issue ? Christian, can you provide a minimal example? Debugging with a breakpoint inside a generated lambda should suffice. I tried the arithmetic xtext example, but have no idea how to actually run the code in the .xtend source. > - Also can you reproduce this issue with pure java project ? No. > - Does your xtext project use maven for dependency management ? No.
@Simeon Does xtext generate source in java or in xtend ? i'm bit confused about the problem. Is the actual problem lies in xtend source debug execution or java debug execution ?
(In reply to Simeon Andreev from comment #3) > Christian, any suggestions here? IMO this is one of the following: > > * JDIStackFrame should report a line number from the generated source. > * The compilation unit should report lines that match the xtext DSL. Without looking into the code, I would expect we generate class files out of xtext DSL, so most likely something with line numbers is messed up in our bytecode generation for lambdas.
(In reply to Gayan Perera from comment #6) > @Simeon Does xtext generate source in java or in xtend ? i'm bit confused > about the problem. Is the actual problem lies in xtend source debug > execution or java debug execution ? As far as I can tell, the problem occurs as follows: 1. Bug 561542 ensures lambda scope variables have human readable names. 2. Bug 560392 uses those names to allow evaluations, using outer scope variables inside the lambda scope. 3.1. xtext generates Java code from some DSL code (xtext, xtend, or something custom as is in our case). The user then seemingly debugs their DSL code, stepping through it. While JDT mostly works on the generated Java code and knows nothing about the DSL code. 3.2. As a result, the fix for bug 561542 is not working (see 1.). 3.3. With the fix for bug 560392 (see 2.), there is now a compile error during the evaluation. In particular, "arg$1" results in a variable name "1" which is invalid. So no evaluation works in the lambda scope, including evaluations that worked prior to the fix for bug 560392. I.e. xtext provides "wrong" JDIStackFrame line information (I'm yet to check how exactly) but until the fix for bug 560392 that "bothered no one". A workaround could be: diff --git a/org.eclipse.jdt.debug/eval/org/eclipse/jdt/internal/debug/eval/ast/engine/ASTEvaluationEngine.java b/org.eclipse.jdt.debug/eval/org/eclipse/jdt/internal/debug/eval/ast/engine/ASTEvaluationEngine.java index fa250018f..8153d896c 100644 --- a/org.eclipse.jdt.debug/eval/org/eclipse/jdt/internal/debug/eval/ast/engine/ASTEvaluationEngine.java +++ b/org.eclipse.jdt.debug/eval/org/eclipse/jdt/internal/debug/eval/ast/engine/ASTEvaluationEngine.java @@ -331,7 +331,7 @@ public class ASTEvaluationEngine implements IAstEvaluationEngine { if (variable instanceof IJavaVariable && !isLambdaOrImplicitVariable(variable)) { IJavaVariable javaVariable = (IJavaVariable) variable; final boolean lambdaField = LambdaUtils.isLambdaField(variable); - String variableName = (lambdaField) ? variable.getName().substring(ANONYMOUS_VAR_PREFIX.length()) : variable.getName(); + String variableName = (lambdaField && variable.getName().startsWith(ANONYMOUS_VAR_PREFIX)) ? variable.getName().substring(ANONYMOUS_VAR_PREFIX.length()) : variable.getName(); if (variableName != null && (!variableName.contains("$") || lambdaField)) { //$NON-NLS-1$ if (!isLocalType(javaVariable.getSignature()) && !names.contains(variableName)) { locals[numLocals] = javaVariable; No idea about an actual fix. Or whether we want a fix or a workaround. (In reply to Andrey Loskutov from comment #7) > Without looking into the code, I would expect we generate class files out of > xtext DSL, so most likely something with line numbers is messed up in our > bytecode generation for lambdas. No idea about this. The line number in JDIStackFrame comes from JDIStackFrame.fLocation, which is the DSL file in this case.
I think also the mentioned two bugs in JDT, 1 fix fixed the problem we had with mixing variables which were not in lambda context. 2 fix Improved the fix to support multi line lambda statements. I don't see a direct connection with this bug and those two bugs provided that you bytecode is generated as @Andrey mentioned or if you Transpiling xtend to java and then compiling into JVM bytecode and expect that the generated lambdas are synched with xtend code.
Please ignore my previous comment since i was referring to the wrong bug
arg$1 variable name is wrong, you will not be able to evaluate a variable name like that with or without bug560392. One situation where the arg$1 variables are described at https://bugs.eclipse.org/bugs/show_bug.cgi?id=560392#c41. Please see if this is the problem you are having. If you get the proper variable names the evaluation should work.
New Gerrit change created: https://git.eclipse.org/r/160832
@sebastian. you might be of better help than me here. please also check the gerrit
Why is this difference in lamnda outer scope variable names in Java -> JVM bytecode and xText -> JVM bytecode. Isn't the problem is in xText -> JVM bytecode where it generate the outerscope variable names different ?
Xtext generates Java. JDT generates bytecode. JDT uses ASM to add line information to bytecode => maybe an asm problem?
Correction: Xtext uses asm to add the Line infos to JDT generatd bytecode
Yes i think we should try to investigate that, because otherwise JDT will need to handle this kind of edge scenarios for all other JVM languages like groovy as well.
(In reply to Christian Dietrich from comment #16) > Correction: > > Xtext uses asm to add the Line infos to JDT generatd bytecode Well then how the variable names are different ? As you say if the java source was compiled using the javac to jvm bytecode, we should get the same val$it as the variable name right ?. The fix seems to handle a situation where we have outer scope variables without the val$ prefix.
(In reply to Simeon Andreev from comment #2) > I see that in JDIStackFrame.LambdaASTVisitor.visit(LambdaExpression), the > following check prevents the update of the lambda variable name to the name > expected by the fix in bug 560392: > > // check if the lineNo fall in lambda region, it can either be single or > multiline lambda body. > if (lineNo < cu.getLineNumber(lambdaExpression.getStartPosition()) > || lineNo > cu.getLineNumber(lambdaExpression.getStartPosition() + > lambdaExpression.getLength())) { > return true; > } > > lineNo is equal to the line in the xtext DSL file, but the compilation unit > is the source file generated by xtext, where the line numbers are different. > So the if expression evaluates to true, and the renaming of the lambda > variables is skipped. This breaks the assumptions in the new code from bug > 560392. Well if this is a issue, isn't that xtext code generation has a problem. IMO i would except this as a bug if you can reproduce it with a plain java project.
@simeon. which are the wrong names?
@simeon: can you reproduce this with pure xtend?
@christian: i would also like if we can see if this is reproducible from pure java project. What is the compiler you are using to compile code into bytecode ?
i dont understand (1) xtext creates a java file (2) eclipse jdt translates this into a .class file (3) Xtext uses asm to (re)write that class file with additional line information
@christian What if we debug the generated source files as a javaproject ? can you reproduce this issue ?
Created attachment 282424 [details] Screenshot from debugging xtext project.
Created attachment 282425 [details] Screenshot from debugging Eclipse + xtext product. // class version 52.0 (52) // access flags 0x21 public class common/ExampleMainFlowData extends ate/ext/prog/javaflow/ReplaceableFlowAlgorithm { // compiled from: ExampleMain.flow @Late/ext/prog/javaflow/OriginInformation;(sourceURI="Test/src/common/ExampleMain.flow", fragment="//@testFlow") @Late/ext/prog/javaflow/SmartestGenerated;() // invisible @Lxoc/dta/internal/annotations/SuppressFBWarnings;() // invisible // access flags 0x19 public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup java/lang/invoke/MethodHandles Lookup // access flags 0x9 public static Z isInSyncWithFile // access flags 0x8 static <clinit>()V ICONST_1 PUTSTATIC common/ExampleMainFlowData.isInSyncWithFile : Z RETURN MAXSTACK = 1 MAXLOCALS = 0 // access flags 0x1 public <init>(Late/ext/prog/javaflow/IAlgorithmFactory;)V L0 ALOAD 0 ALOAD 1 INVOKESPECIAL ate/ext/prog/javaflow/ReplaceableFlowAlgorithm.<init>(Late/ext/prog/javaflow/IAlgorithmFactory;)V RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE algorithmFactoryPool Late/ext/prog/javaflow/IAlgorithmFactory; L0 L1 1 MAXSTACK = 2 MAXLOCALS = 2 // access flags 0x1 public isInSyncWithFile()Z L0 GETSTATIC common/ExampleMainFlowData.isInSyncWithFile : Z IRETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 MAXSTACK = 1 MAXLOCALS = 1 // access flags 0x4 protected initializeParameters(Late/ext/prog/javaflow/IFlow;)V L0 RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE initializedFlow Late/ext/prog/javaflow/IFlow; L0 L1 1 MAXSTACK = 0 MAXLOCALS = 2 // access flags 0x1 public initializeTestSuites(Late/ext/prog/javaflow/IFlow;Ljava/util/Map;)V L0 ALOAD 0 ALOAD 1 ALOAD 2 LDC "Functional" LDC "common.ExampleMain____FunctionalSuiteData" INVOKEVIRTUAL common/ExampleMainFlowData.initializeTestSuite(Late/ext/prog/javaflow/IFlow;Ljava/util/Map;Ljava/lang/String;Ljava/lang/String;)V RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE initializedFlow Late/ext/prog/javaflow/IFlow; L0 L1 1 LOCALVARIABLE currentSuites Ljava/util/Map; L0 L1 2 MAXSTACK = 5 MAXLOCALS = 3 // access flags 0x1 public initializeSubflows(Late/ext/prog/javaflow/IFlow;Ljava/util/Map;)V L0 RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE initializedFlow Late/ext/prog/javaflow/IFlow; L0 L1 1 LOCALVARIABLE currentSubflows Ljava/util/Map; L0 L1 2 MAXSTACK = 0 MAXLOCALS = 3 // access flags 0x1 public initializeCharacterizations(Late/ext/prog/javaflow/IFlow;Ljava/util/Map;)V L0 RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE initializedFlow Late/ext/prog/javaflow/IFlow; L0 L1 1 LOCALVARIABLE currentChars Ljava/util/Map; L0 L1 2 MAXSTACK = 0 MAXLOCALS = 3 // access flags 0x1 public executeTestflow(Late/ext/prog/javaflow/IFlow;)V L0 LINENUMBER 14 L0 ALOAD 1 LDC "Functional" INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.getSuite(Ljava/lang/String;)Late/ext/prog/javaflow/ISuite; INVOKEINTERFACE ate/ext/prog/javaflow/ISuite.execute()V L1 LINENUMBER 16 L1 ALOAD 1 INVOKEDYNAMIC run(Late/ext/prog/javaflow/IFlow;)Ljava/lang/Runnable; [ // handle kind 0x6 : INVOKESTATIC java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: ()V, // handle kind 0x6 : INVOKESTATIC common/ExampleMainFlowData.lambda$0(Late/ext/prog/javaflow/IFlow;)V, ()V ] ASTORE 2 L2 LINENUMBER 19 L2 ALOAD 1 INVOKEDYNAMIC run(Late/ext/prog/javaflow/IFlow;)Ljava/lang/Runnable; [ // handle kind 0x6 : INVOKESTATIC java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: ()V, // handle kind 0x6 : INVOKESTATIC common/ExampleMainFlowData.lambda$1(Late/ext/prog/javaflow/IFlow;)V, ()V ] ASTORE 3 L3 LINENUMBER 16 L3 ALOAD 1 LDC "Functional" INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.getSuite(Ljava/lang/String;)Late/ext/prog/javaflow/ISuite; LDC "pass" LDC Lxoc/dta/datatypes/MultiSiteBoolean;.class INVOKEINTERFACE ate/ext/prog/javaflow/ISuite.getParameter(Ljava/lang/String;Ljava/lang/Class;)Ljava/lang/Object; CHECKCAST xoc/dta/datatypes/MultiSiteBoolean ALOAD 2 ALOAD 3 INVOKESTATIC ate/ext/prog/javaflow/MultiSiteIf.run(Lxoc/dta/datatypes/MultiSiteBoolean;Ljava/lang/Runnable;Ljava/lang/Runnable;)V L4 LINENUMBER 16 L4 ALOAD 1 INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.hasNoActiveSites()Z IFEQ L5 RETURN L5 FRAME APPEND [java/lang/Runnable java/lang/Runnable] RETURN L6 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L6 0 LOCALVARIABLE it Late/ext/prog/javaflow/IFlow; L0 L6 1 LOCALVARIABLE _function_1 Ljava/lang/Runnable; L2 L6 2 LOCALVARIABLE _function_2 Ljava/lang/Runnable; L3 L6 3 MAXSTACK = 3 MAXLOCALS = 4 // access flags 0x1 public dependsOnTPVariable(Ljava/lang/String;)Z L0 ICONST_0 IRETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE tpVariable Ljava/lang/String; L0 L1 1 MAXSTACK = 1 MAXLOCALS = 2 // access flags 0x1 public getSetupHash()Ljava/lang/String; L0 LDC "19ef21fb" ARETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 MAXSTACK = 1 MAXLOCALS = 1 // access flags 0x1 public debugHook(Late/ext/prog/javaflow/IFlow;)V L0 LINENUMBER 13 L0 RETURN L1 LOCALVARIABLE this Lcommon/ExampleMainFlowData; L0 L1 0 LOCALVARIABLE it Late/ext/prog/javaflow/IFlow; L0 L1 1 MAXSTACK = 0 MAXLOCALS = 2 // access flags 0x100A private static synthetic lambda$0(Late/ext/prog/javaflow/IFlow;)V L0 LINENUMBER 17 L0 ALOAD 0 LDC "Yeah!" INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.println(Ljava/lang/Object;)V L1 LINENUMBER 18 L1 ALOAD 0 ICONST_1 INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.addBin(I)V RETURN MAXSTACK = 2 MAXLOCALS = 1 // access flags 0x100A private static synthetic lambda$1(Late/ext/prog/javaflow/IFlow;)V L0 LINENUMBER 20 L0 ALOAD 0 LDC "OMG!" INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.println(Ljava/lang/Object;)V L1 LINENUMBER 21 L1 ALOAD 0 ICONST_2 INVOKEINTERFACE ate/ext/prog/javaflow/IFlow.addBin(I)V RETURN MAXSTACK = 2 MAXLOCALS = 1 }
@Simeon this not the source code right, this is bytecode (i'm not a expert on ByteCode stuff). So the flow file directly get generated into a class file ? or does it get traspilled into .java source and then compiled to a .class file.
@Gayan nope. the bytecode is created by jdt we just use ASM to wrote JSR-45 information to the class files afterwards
i dont know if this ibuild is anywhere. in my WS (using 4.16 i-builds) i am getting val$it
@christian @simeon I'm not a expert in JDT inner workings, but from my understanding while working in the JDT area i can say the following. If we take the java file that you have in your screenshot and try to debug it, i think you should be able to do it if you have all the required dependencies. Do we see a problem in trying to debug addBin(l) method ? from your screenshots it seems the JDT logic is correct at JDIStackFrame when looking at your generated code. I think your sourcemapping might have a issue, because for the addBin(l) in your flow DSL which is line 18 should translate to like 51 in the java source code to get the correct variable information from the JDT. So isn't the problem in this line mapping ?
@Gayan the line numbers relate to the flow file (See JSR-45)
Let me discuss this with Andrey tomorrow.
@Simeon good, and also its good to get some input from @Sarika on this since she know both of those bugs well in and out.
@christian Seems like you are rewrting the the orginal java source line numbers with flow file line numbers ? did i got you right ?
Jumping into the discussion, I try to add a little bit of background information about debugging Xtext DSLs. As Christian already mentioned, Xtext is used to generate Java source code. That code is compiled by JDT in the context of an IJavaProject with the settings that apply to the regular compilation of all the source code in that Java project. E.g. if you untick some of the options about the classfile generation, the compiled code will not have information about parameter names, local var names etc. After the class file was written by JDT, Xtext uses ASM to read the classfile and add more debug information. Depending on the configuration of the project, Xtext will either add an aditional stratum to the classfile, that points to the original DLS file(s) that were used to create the Java source code (JSR 45, same technology that is used to debug JSP files), or alternatively the regular Java source information will be replaced by the DSL source line numbers. If the second option is chosen, there additional configuration flags, e.g. to hide some local vars that are synthesised during the Java source code generation, but variable names like 'it' will not be changed. Things to watch out for when tracking this issue down: Is the project configured correctly, e.g. do we see Java local vars when debugging plain Java code? Are parameter names kept in the class file correctly? Does the JSR45 support in JDT work with lambda expressions? Do the line numbers in the stratum match the real line numbers in Java and the DSL file? Does debugging / evaluating expression work, if you chose 'Show source -> Java' in the debugger on the DSL stack frame?
@Sebastian thanks good insight into this. I have some 3 years back experience working on xtext dsl and working with xtend to writing the code generator for my dsl. Well i think the problem here is when we ask for CompilationUnit here in JDIStackFrame.visit method, we get line number information for java source code, not for the DSL source code which is represented by lineNo. So i think root cause for this issue is in here. We should be able to either convert the lineNo to javasource code line number which is 51 or we should be able to get the cu.getLineNumber for given offset from the DSL file perspective. Therefore i think we should not try to fix this in ASTEvaluationEngine, rather we should try to fix the problems which doesn't provide the variable names as shown in the screenshots. What do you guys think ?
OK, once again, hopefully this time the complete information. 1. We use xtext to define a DSL. The user writes DSL. 2. xtext generates Java code from the DSL, in the bug case it generates a lambda expression. 3. We compile the Java code with Eclipse, to byte code. 4. xtext adjusts the byte code ("without telling Eclipse"), to add line information that matches the DSL. 5. During debugging, JDT asks the JVM for line numbers and "sees" the line numbers set by xtext in step 4.. 6.1. The code that fixes bug 561542 uses the adjusted line numbers (step 4.) and an ASTParser on the generated Java code (step 2.). 6.2. The lambda expression in the generated Java code does not match (line-wise) the DSL code. So the fix for bug 561542 does nothing. 6.3. So we have "arg$1" and so on, instead of "val$it" and so on, as lambda variables on the stack frame. 7.1. The fix for bug 560392 assumes step 6. went well and removes the "arg$" prefix, "thinking" it removes the "val$" prefix. 7.2. The remaining variable name is "1", which is illegal. 7.3. Any evaluation within the lambda expression (generated in step 2.) results in an error, since the evaluation scope contains compile problems. With https://git.eclipse.org/r/#/c/160832/, the fix for bug 560392 is applied only if the fix for bug 561542 "worked". If not, the synthetic variable names are used, which prevents using outer scope (visible) variables to be used in expressions (but only for the very specific conditions, xtext + generated lambda). We don't know how to fix xtext here, we are only adjusting ASTEvaluationEngine to not produce a broken context for evaluations (so that evaluations that worked before the fix for bug 560392 continue to work).
(In reply to Simeon Andreev from comment #38) > hopefully this time the complete information. Is (4) custom logic or Xtext default debugging information enhancement. If it is the latter, do you use JSR45 capabilities, or do you use the mode that replaces the Java line information.
@Sebastian. we use TraceAsPrimarySourceInstaller thus this should be reproducible with Xtend (when configured respectively) as well
(In reply to Simeon Andreev from comment #38) > With https://git.eclipse.org/r/#/c/160832/, the fix for bug 560392 is > applied only if the fix for bug 561542 "worked". If not, the synthetic > variable names are used, which prevents using outer scope (visible) > variables to be used in expressions (but only for the very specific > conditions, xtext + generated lambda). I see your point now @Simeon, sorry if i was a PITA upto now :(. The problem is the fix in bug560392 assumed that lambda variables are always resolved. You fix might solve future issues for other jvm languages as well. > We don't know how to fix xtext here, we are only adjusting > ASTEvaluationEngine to not produce a broken context for evaluations (so that > evaluations that worked before the fix for bug 560392 continue to work). Can you see if you get both the line numbers here at org.eclipse.jdt.internal.debug.core.model.JDIStackFrame.setLambdaVariableNames(IJavaValue, ObjectReference) try { allLineLocations = getUnderlyingMethod().allLineLocations(); int lineNo = allLineLocations.get(0).lineNumber(); cu.accept(new LambdaASTVisitor(false, underlyingThisObject, getUnderlyingMethod().isStatic(), cu, lineNo)); } catch (AbsentInformationException e) { e.printStackTrace(); } Because we only get the first element.
@Simeon have you check about my previous comment ? By the way do you keep both line number information dsl and java both ?
As Sebastian said: In Xtend: you can decide which mode you want to have - SMAP (whichb basically keeps the Java Lines and adds additional information about source) - replacing the java lines with dsl lines - in our DSL we do the second
@Christian should we try to keep both for this kind of resolutions ? Because there might be a need to read some variables in inspections which lambda fields in compiled lambda classes
if i understand JSR-45 correct both are valid ways and should be supported
(In reply to Gayan Perera from comment #42) > [..] > Can you see if you get both the line numbers here at > org.eclipse.jdt.internal.debug.core.model.JDIStackFrame. > setLambdaVariableNames(IJavaValue, ObjectReference) > > try { > allLineLocations = getUnderlyingMethod().allLineLocations(); > int lineNo = allLineLocations.get(0).lineNumber(); > cu.accept(new LambdaASTVisitor(false, underlyingThisObject, > getUnderlyingMethod().isStatic(), cu, lineNo)); > } catch (AbsentInformationException e) { > e.printStackTrace(); > } > > Because we only get the first element. Here are the locations I see when debugging: [sourcename: ExampleMain.flow, line: 17, method: lambda$0, sourcename: ExampleMain.flow, line: 18, method: lambda$0]
@Sarika if we add multiple line info in class level is there a way we can retrieve that here and read the java line numbers in this kind of a situation to match against the AST ?
(In reply to Gayan Perera from comment #48) > @Sarika if we add multiple line info in class level is there a way we can > retrieve that here and read the java line numbers in this kind of a > situation to match against the AST ? Not sure what do you mean by that, but as proposed by Sebastian we simply shouldn't try to resolve anything from Java code if the class file says it belongs to another language.
(In reply to Andrey Loskutov from comment #49) > (In reply to Gayan Perera from comment #48) > > @Sarika if we add multiple line info in class level is there a way we can > > retrieve that here and read the java line numbers in this kind of a > > situation to match against the AST ? > > Not sure what do you mean by that, but as proposed by Sebastian we simply > shouldn't try to resolve anything from Java code if the class file says it > belongs to another language. My suggestion was whether we can get both line numbers that is java file and flow file. So that we can choose the correct line cordinate to resolve the variables. Otherwise we might missed some variables info for the flow file debug seasion isn’t it ?
Class file Line number is maintained by JDT Core, I don't think we can maintain xtext line number as well. I agree, that we should check for java file and then only use the line number.
@simeon. as said before. we can if we use the smap mode (this is the default) so if you create a xtend file with lamba this is basically done
(In reply to Simeon Andreev from comment #1) > Sarika, any idea why we would have a variable named "arg$1" and not "val$it" > while in the lambda for "_functional_1" (see the snippet in the description)? arg was used as it was the argument to the lambda method.
(In reply to Sarika Sinha from comment #51) > Class file Line number is maintained by JDT Core, I don't think we can > maintain xtext line number as well. > I agree, that we should check for java file and then only use the line > number. Then ideally the jdt debugger should not be used to debugging other languages right ? Because here we are just using because it just works. I think we should have way of using something like source maps to map flow file line number to java line number as a extension so that a jvm language can write such extension What do you all think ?
(In reply to Gayan Perera from comment #54) > (In reply to Sarika Sinha from comment #51) > > Class file Line number is maintained by JDT Core, I don't think we can > > maintain xtext line number as well. > > I agree, that we should check for java file and then only use the line > > number. > > Then ideally the jdt debugger should not be used to debugging other > languages right ? Because here we are just using because it just works. No, debugger can be used but it should not blindly assume the *source* corresponding to the loaded type is always written in *Java* language. We have various languages running on JVM, all using common *class* format, and as you can see on the screenshot, we can perfectly debug such non-Java languages with Java debugger. I will push updated patch in a moment. Note: the loaded class says unfortunately that it is from "Java" stratum, but fortunately the source name is clearly not Java: ExampleMain.flow :-)
> > Note: the loaded class says unfortunately that it is from "Java" stratum, > but fortunately the source name is clearly not Java: ExampleMain.flow :-) May be something to fix in xtext ASM modifications ?
> No, debugger can be used but it should not blindly assume the *source* > corresponding to the loaded type is always written in *Java* language. We > have various languages running on JVM, all using common *class* format, and > as you can see on the screenshot, we can perfectly debug such non-Java > languages with Java debugger. > So that means we will not try resolved lambda variables if the source is not java is it ?
(In reply to Gayan Perera from comment #57) > So that means we will not try resolved lambda variables if the source is not > java is it ? Please check new patch set https://git.eclipse.org/r/#/c/160832/3.
@Andrey Looks good on handling the issue at hand, but if we don't resolve the variable names, can those variables be inspected in variables view, on inspections in debugshell when debugging a DSL file ? @Simeon, @Christian Don't you have such requirments,
in general there is the requirement to resolve
I should have hold my horses :) The answer to my question is in your commit message, So if i understand correctly, XText can provide a JDIStackFrame implementation for flow files right @Andrey ?
(In reply to Gayan Perera from comment #59) > @Andrey > Looks good on handling the issue at hand, but if we don't resolve the > variable names, can those variables be inspected in variables view, on > inspections in debugshell when debugging a DSL file ? Sure, they can, but instead of "val$hello" you will see "arg$1" (same as provided by class files). Please note: in *Java* debugger we have no clues about captured lambda variable names if the names aren't coming from *Java* source code. Resolving that info is beyond of JDT capabilities.
This bug has more impact than I originally assumed. I cannot evaluate expressions in the stack frame from bug 562220: java.lang.NullPointerException at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.lambda$1(CompatibilityPart.java:122) When I try inspect "reference.getPage()" there is an error "evaluations must contain a valid expression". In the Error Log I see: eclipse.buildId=4.16.0.I20200409-0200 java.version=1.8.0_252-ea java.vendor=Oracle Corporation BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US Command-line arguments: -os linux -ws gtk -arch x86_64 -data /home/sandreev/development_workspaces/contributor_workspace/ws/ org.eclipse.debug.core Warning Thu Apr 16 16:18:07 CEST 2020 Compile error during code evaluation: Syntax error, insert "... VariableDeclaratorId" to complete FormalParameter eclipse.buildId=4.16.0.I20200409-0200 java.version=1.8.0_252-ea java.vendor=Oracle Corporation BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US Command-line arguments: -os linux -ws gtk -arch x86_64 -data /home/sandreev/development_workspaces/contributor_workspace/ws/ org.eclipse.debug.core Warning Thu Apr 16 16:18:07 CEST 2020 Compile error during code evaluation: Syntax error on token "1", invalid VariableDeclaratorId This is a pure Java example, unless I'm mistaken. The lambda in question is set on: org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.objectSetHandler I assume the fix for bug 561542 doesn't work in this case.
Created attachment 282476 [details] Cannot evaluate in member CompatibilityPart.objectSetHandler initialized with a lambda.
(In reply to Sarika Sinha from comment #51) > Class file Line number is maintained by JDT Core, I don't think we can > maintain xtext line number as well. > I agree, that we should check for java file and then only use the line > number. Line number info can also be obtained from editor, vertical rule information. To understand the JSR 45 debugger support usage, checkout /org.eclipse.jdt.ui.examples.javafamily
(In reply to Sarika Sinha from comment #65) > (In reply to Sarika Sinha from comment #51) > > Class file Line number is maintained by JDT Core, I don't think we can > > maintain xtext line number as well. > > I agree, that we should check for java file and then only use the line > > number. > > Line number info can also be obtained from editor, vertical rule > information. To understand the JSR 45 debugger support usage, checkout > /org.eclipse.jdt.ui.examples.javafamily Sarika, we have no bandwidth (and serious interest in JSR 45 support in general) to work on that. In general our DSL based debugging works just fine with the current JDT (we find time to time some bugs here and there, but nothing related to the current problem with JSR 45). I've added real word test case to the patch that shows the regression from bug 560392 that can be reproduced with plain Java code. The patch fixes this regression and also as a side effect fixes our DSL related regression. I honestly can't spend more time on that, and would like to merge it. Would you accept that?
(In reply to Andrey Loskutov from comment #66) > (In reply to Sarika Sinha from comment #65) > > (In reply to Sarika Sinha from comment #51) > > > Class file Line number is maintained by JDT Core, I don't think we can > > > maintain xtext line number as well. > > > I agree, that we should check for java file and then only use the line > > > number. > > > > Line number info can also be obtained from editor, vertical rule > > information. To understand the JSR 45 debugger support usage, checkout > > /org.eclipse.jdt.ui.examples.javafamily > > Sarika, we have no bandwidth (and serious interest in JSR 45 support in > general) to work on that. In general our DSL based debugging works just fine > with the current JDT (we find time to time some bugs here and there, but > nothing related to the current problem with JSR 45). > > I've added real word test case to the patch that shows the regression from > bug 560392 that can be reproduced with plain Java code. The patch fixes this > regression and also as a side effect fixes our DSL related regression. I > honestly can't spend more time on that, and would like to merge it. Would > you accept that? Yes, I am fine with the current change. Gerrit having issue, so could not put +2 there. As Gayan had asked I just wanted to put the information so that it can be used to understand the code base.
(In reply to Sarika Sinha from comment #67) > Yes, I am fine with the current change. Gerrit having issue, so could not > put +2 there. Thanks, I see +2 but I can't merge. Do we have a bug for Gerrit issues?
(In reply to Andrey Loskutov from comment #68) > (In reply to Sarika Sinha from comment #67) > > Yes, I am fine with the current change. Gerrit having issue, so could not > > put +2 there. > > Thanks, I see +2 but I can't merge. Do we have a bug for Gerrit issues? I was able to merge, git commit is https://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=ac9edad720070520bfb97887e88b15145efef9bb . Gerrit bug is bug 562301.
Verified patch in our product: our tests are green again.
