Bug 359943 - invokedynamic in generated class file is not correctly recognized by the eclipse compiler
Summary: invokedynamic in generated class file is not correctly recognized by the ecli...
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.8   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: 3.8 M3   Edit
Assignee: Olivier Thomann CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-10-05 00:57 EDT by Sebastian Sickelmann CLA
Modified: 2011-10-25 10:17 EDT (History)
5 users (show)

See Also:
amj87.iitr: review+


Attachments
Proposed fix (60.25 KB, patch)
2011-10-05 15:19 EDT, Olivier Thomann CLA
no flags Details | Diff
Proposed fix (62.59 KB, patch)
2011-10-05 16:00 EDT, Olivier Thomann CLA
no flags Details | Diff
Sample of G.class that is using some of the new constant pool entries (311 bytes, application/octet-stream)
2011-10-05 16:57 EDT, Olivier Thomann CLA
no flags Details
Class file that should contain the methodtype constant pool (324 bytes, application/java-vm)
2011-10-06 00:58 EDT, Sebastian Sickelmann CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sebastian Sickelmann CLA 2011-10-05 00:57:53 EDT
Build Identifier: 20110916-0149

An invokedynamic instruction in a generated classfile is not resolved correct by the eclipse compiler and the "Class File Editor"-view. I uploaded a demo project to my github-incubator.

Reproducible: Always

Steps to Reproduce:
1. Download eclipse project under jdk/invokedynamic/helloDyn from git://github.com/picpromusic/incubator.git (for a quick look use this url:https://github.com/picpromusic/incubator/tree/master/jdk/invokedynamic/helloDyn)
2. Install (http://forge.ow2.org/projects/asm/) and install it as eclipse library (name asm4) to complete classpath of the project. I used asm 4.0 RC1
3. Compile with eclipse and start class M (output should be START\nDONE\nEND\n)
4. Start class GEN (this generated a class G in directory gen which is up most on the classpath)
5. Start class M (without any refresh/recompile on the project) (output should be (START\nBootstrap\nDONE\nEND)
6. Refresh project / Recompile Project (now there is a compile error in M, the class G cannot be resolved)
7. Double click on gen/G.class doesn't show up no byte code instructions (maybe it has the same reason as the unresolveable G)
Comment 1 Ayushman Jain CLA 2011-10-05 09:39:38 EDT
(In reply to comment #0)
> 2. Install (http://forge.ow2.org/projects/asm/) and install it as eclipse
> library (name asm4) to complete classpath of the project. I used asm 4.0 RC1

I downloaded asm-4.0RC1.jar, but when i add that to the classpath I still see classpath errors. 

Project 'helloDyn' is missing required Java project: 'asm3'
Project 'helloDyn' is missing required library: 'gen'

Am i missing something?
Comment 2 Sebastian Sickelmann CLA 2011-10-05 10:39:14 EDT
(In reply to comment #1)
> (In reply to comment #0)
> > 2. Install (http://forge.ow2.org/projects/asm/) and install it as eclipse
> > library (name asm4) to complete classpath of the project. I used asm 4.0 RC1
> 
> I downloaded asm-4.0RC1.jar, but when i add that to the classpath I still see
> classpath errors. 
> 
> Project 'helloDyn' is missing required Java project: 'asm3'
> Project 'helloDyn' is missing required library: 'gen'
> 
> Am i missing something?

I am sorry. Forgot to push my classpath changes, now it should use asm4 as System-library. If you want to add asm-4.0RC1.jar to the classpath by your own, then you must remove the asm4 library entry in classpath.

Please pull actual version.

the gen-directory was deleted by git, because it was empty. Now i put a .txt file in the dir.

Sorry. It was my fault.
Comment 3 Olivier Thomann CLA 2011-10-05 10:54:58 EDT
Reproduced it by manually updating the classpath. I am working on adding support for the new constant pool entries.
Comment 4 Olivier Thomann CLA 2011-10-05 15:19:47 EDT
Created attachment 204625 [details]
Proposed fix

You might have to remove the first two segments of the patch to be able to apply it.

With this patch, G.class can now be disassembled:
//  (version 1.7 : 51.0, no super bit)
public class G {
  
  // Method descriptor #6 ()V
  // Stack: 0, Locals: 0
  public static void call();
    0  invokedynamic 0 dyn() : void [17]
    5  return

Bootstrap methods:
  0 : # 13 arguments: {#14}
}

It looks better when using the disassembler from the command line:
//  (version 1.7 : 51.0, no super bit)
public class G {
  Constant pool:
    constant #1 utf8: "G"
    constant #2 class: #1 G
    constant #3 utf8: "java/lang/Object"
    constant #4 class: #3 java/lang/Object
    constant #5 utf8: "call"
    constant #6 utf8: "()V"
    constant #7 utf8: "M"
    constant #8 class: #7 M
    constant #9 utf8: "bootstrap"
    constant #10 utf8: "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;"
    constant #11 name_and_type: #9.#10 bootstrap (Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;
    constant #12 method_ref: #8.#11 M.bootstrap (Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;
    constant #13 method handle: invokestatic (6) #12 
    constant #14 integer: 1
    constant #15 utf8: "dyn"
    constant #16 name_and_type: #15.#6 dyn ()V
    constant #17 invoke dynamic: #0 #16 dyn ()V
    constant #18 utf8: "Code"
    constant #19 utf8: "BootstrapMethods"
  
  // Method descriptor #6 ()V
  // Stack: 0, Locals: 0
  public static void call();
    0  invokedynamic 0 dyn() : void [17]
    5  return

Bootstrap methods:
  0 : # 13 arguments: {#14}
}

And the reference to G.call() is not longer seen as unresolved.

Ayushman, please review.
Comment 5 Olivier Thomann CLA 2011-10-05 15:51:11 EDT
According to Jay, that patch doesn't apply well. Working on a new patch.
Comment 6 Olivier Thomann CLA 2011-10-05 16:00:19 EDT
Created attachment 204629 [details]
Proposed fix

This patch should apply, but it might report whitespaces warnings.
Comment 7 Olivier Thomann CLA 2011-10-05 16:34:43 EDT
Daniel,

I was not aware of any particular changes in the class file format for Java 7. New constant pool entries have been added and a new attribute (Bootstrap Methoew) as well.
The patch adds the missing pieces and tried to prevent API breakage. I added a new interface IConstantPoolEntry2 to handle the new entries.

Let me know what you think. The given test case can now be handled without an issue. I am not how frequent .class files with these new elements can be found. So for now I would only fix this inside master and backport it to 3.7.2 only if necessary.
Comment 8 Olivier Thomann CLA 2011-10-05 16:55:10 EDT
Sebastian, could you please provide an example that would end up using the MethodType constant pool entry to validate the new support?
Thanks.
Comment 9 Olivier Thomann CLA 2011-10-05 16:57:21 EDT
Created attachment 204639 [details]
Sample of G.class that is using some of the new constant pool entries

This .class file cannot be properly disassembled using the disassembler inside master. It works fine with the patch.
Comment 10 Sebastian Sickelmann CLA 2011-10-05 23:42:25 EDT
(In reply to comment #7)
> Daniel,
> 
> I was not aware of any particular changes in the class file format for Java 7.
> New constant pool entries have been added and a new attribute (Bootstrap
> Methoew) as well.
> The patch adds the missing pieces and tried to prevent API breakage. I added a
> new interface IConstantPoolEntry2 to handle the new entries.
> 
> Let me know what you think. The given test case can now be handled without an
> issue. I am not how frequent .class files with these new elements can be found.
> So for now I would only fix this inside master and backport it to 3.7.2 only if
> necessary.

I am not quite sure how often this will be used in the nearest future but almost any dynamic language that runs on the jvm is actually porting to invoke dynamic. And for interoperability it would be nice to have this in 3.7.2.

Not be able to build against a library that is created with ex. "jruby for jvm7" would a real blocker in the future.
Comment 11 Sebastian Sickelmann CLA 2011-10-06 00:56:32 EDT
(In reply to comment #9)
> Created attachment 204639 [details]
> Sample of G.class that is using some of the new constant pool entries
> 
> This .class file cannot be properly disassembled using the disassembler inside
> master. It works fine with the patch.

(In reply to comment #8)
> Sebastian, could you please provide an example that would end up using the
> MethodType constant pool entry to validate the new support?
> Thanks.

For this i am unfortuantly forces to refactor the project. So that G now is in package test. I pushed these changed to my github-incubator.

Please pull the changes.

I will upload a shrinked classfile (this should be contain the MethodType constant pool, but i am not sure it does).
Comment 12 Sebastian Sickelmann CLA 2011-10-06 00:58:57 EDT
Created attachment 204649 [details]
Class file that should contain the methodtype constant pool

Don't know if it realy contains the constant pool. And i don't know why a shrinked (ASM-Library Shrinker-Class) version of the class is actually bigger that the original. Maybe because of another constant-pool.
Comment 13 Dani Megert CLA 2011-10-06 01:53:23 EDT
Let's keep this on the waiting list for 3.7.2. Depending on whether we get more reports during the next weeks we can include it.
Comment 14 Olivier Thomann CLA 2011-10-06 16:21:00 EDT
(In reply to comment #12)
> Don't know if it realy contains the constant pool. And i don't know why a
> shrinked (ASM-Library Shrinker-Class) version of the class is actually bigger
> that the original. Maybe because of another constant-pool.
Thanks. It does contain the MethodType constant pool entry.
//  (version 1.2 : 46.0, no super bit)
public class test.G {
  Constant pool:
    constant #1 integer: 1
    constant #2 utf8: "()V"
    constant #3 utf8: "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;"
    constant #4 utf8: "Code"
    constant #5 utf8: "bootstrap"
    constant #6 utf8: "call"
    constant #7 utf8: "dyn"
    constant #8 utf8: "java/lang/Object"
    constant #9 utf8: "test/G"
    constant #10 utf8: "test/M"
    constant #11 class: #8 java/lang/Object
    constant #12 class: #9 test/G
    constant #13 class: #10 test/M
    constant #14 name_and_type: #5.#3 bootstrap (Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;
    constant #15 name_and_type: #7.#2 dyn ()V
    constant #16 method_ref: #13.#14 test/M.bootstrap (Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/CallSite;
    constant #17 method handle: invokestatic (6) #16 
    constant #18 invoke dynamic: #0 #15 dyn ()V
    constant #19 method type: #7 dyn
    constant #20 utf8: "BootstrapMethods"
  
  // Method descriptor #2 ()V
  // Stack: 0, Locals: 0
  public static void call();
    0  invokedynamic 0 dyn() : void [18]
    5  return

Bootstrap methods:
  0 : # 17 arguments: {#1}
}

I'll commit the code to origin/master to ease Ayushman's review as the patch has some whitespace issues.
Ayushman, I'll update the buildnotes with the new APIs once your review is done.
Comment 15 Ayushman Jain CLA 2011-10-07 05:45:42 EDT
Patch looks good. Verified against JVM spec (Java 7 edition).

Olivier, just like we added tests for the new Stackmap attribute in java6, can we also add a few for the new Bootstrap methods attribute?
Comment 16 Olivier Thomann CLA 2011-10-07 08:07:46 EDT
I think we should use the given G.class ans open them using the disassembler for regression tests.
Comment 17 Stephan Herrmann CLA 2011-10-08 14:14:55 EDT
(In reply to comment #14)
> I'll commit the code to origin/master ...

could this commit be responsible for the regressions in builds >= 20111007?

We get a header like
// Compiled from X.java (version 1.6 : 50.0, super bit)\n
where it is not expected.

I couldn't exactly see what would have caused this to change, did anything
in the patch change the disassemble mode??
Comment 18 Olivier Thomann CLA 2011-10-09 22:26:07 EDT
(In reply to comment #17)
> could this commit be responsible for the regressions in builds >= 20111007?
It was responsible for the 14 failures. I cannot explain all the failures on Windows XP 1.6.
Should be fixed either in this N-build or the next one.
Comment 19 Ayushman Jain CLA 2011-10-10 01:29:25 EDT
(In reply to comment #18)
> (In reply to comment #17)
> > could this commit be responsible for the regressions in builds >= 20111007?
> It was responsible for the 14 failures. I cannot explain all the failures on
> Windows XP 1.6.
> Should be fixed either in this N-build or the next one.

Filed bug 360379 to track this.
Comment 20 Ayushman Jain CLA 2011-10-12 10:13:24 EDT
Added a regression test using the G.class file.
Released in HEAD via Commit id 18358d3eb05898fd25311c84d0aaa48b90eefeb3
Comment 21 Stephan Herrmann CLA 2011-10-25 10:17:54 EDT
Verified for 3.8 M3 using build N20111022-2000