Bug 410325 - [1.7][compiler] Generified method override different between javac and eclipse compiler
Summary: [1.7][compiler] Generified method override different between javac and eclips...
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.3   Edit
Hardware: PC Mac OS X
: P3 normal (vote)
Target Milestone: 4.3.1   Edit
Assignee: Stephan Herrmann CLA
QA Contact:
URL:
Whiteboard: To be verified for 4.4 M2
Keywords:
: 411811 415600 416917 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-06-10 07:51 EDT by Martijn Dashorst CLA
Modified: 2013-09-10 10:12 EDT (History)
6 users (show)

See Also:
daniel_megert: pmc_approved+


Attachments
Class that compiles to different override behaviors between javac and eclipse (4.3RC3) (952 bytes, text/plain)
2013-06-10 07:51 EDT, Martijn Dashorst CLA
no flags Details
Complete project that shows the behavior from comment #0 (10.77 KB, application/octet-stream)
2013-06-11 06:38 EDT, Martijn Dashorst CLA
no flags Details
Eclipse configuration (412.63 KB, text/plain)
2013-06-11 06:42 EDT, Martijn Dashorst CLA
no flags Details
patch under test (5.79 KB, patch)
2013-08-22 16:02 EDT, Stephan Herrmann CLA
no flags Details | Diff
Another case, where the solution doesn't work yet. (7.20 KB, application/x-zip-compressed)
2013-09-10 06:57 EDT, Tomasz Stanczak CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martijn Dashorst CLA 2013-06-10 07:51:45 EDT
Created attachment 232173 [details]
Class that compiles to different override behaviors between javac and eclipse (4.3RC3)

When compiling the attached java file using javac and eclipse 4.2, no compilation errors occur. Using Eclipse 4.3RC3 (4.3M6 also has this issue) gives a compilation error:

The type Main.SubSub must implement the inherited abstract method Main.Base<Main.F3>.foo(Main.F3)	Main.java(42)

Removing the 'abstract' keyword from the Main#Sub class also removes the compilation error, as does implementing a `foo(F3 f)` method. The Eclipse generated byte code results in different runtime behavior when compared to the output generated by the javac compiler.

An implementation of the foo(F3) method in the SubSub type (with the Sub type remaining abstract):


	@Override
	public void foo(F3 bar) {
		System.out.println(getClass().getSimpleName() + ": F3 + "
				+ bar.getClass().getSimpleName());
		bar.bar();
	}

The expected result as given by javac/java

    SubSub: F3 + F3
    bar in F3
    SubSub: F3 + F3
    bar in F3
    SubSub: F3 + F3
    bar in F3
    SubSub: F2 + F2

The result as given by the eclipse compiler:

    SubSub: F3 + F3
    bar in F3
    SubSub: F2 + F3
    SubSub: F3 + F3
    bar in F3
    SubSub: F2 + F2

As you can see the call to 'sub2.foo(f3);' performed in Main#main(), line 11 gives a strange result, whereas the call in line 12 does call the correct function (giving the expected result when compared to javac)

Commandline java version:

java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b12)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
Comment 1 Jay Arthanareeswaran CLA 2013-06-11 04:47:08 EDT
Stephan, looks like a candidate for 4.3.1. Can you please take a look?
Comment 2 Stephan Herrmann CLA 2013-06-11 06:01:15 EDT
ack
Comment 3 Stephan Herrmann CLA 2013-06-11 06:23:36 EDT
I can reproduce the different program output when compiled with ecj vs. javac.

(In reply to comment #0)
> Using Eclipse 4.3RC3 (4.3M6 also has this issue)
> gives a compilation error:
> 
> The type Main.SubSub must implement the inherited abstract method
> Main.Base<Main.F3>.foo(Main.F3)	Main.java(42)

This I cannot reproduce. All versions >= 3.6 (!) compile the program without
any error. Earlier versions report:

The method foo(Main.F3) of type Main.SubSub must override or implement a supertype method


Note, that I could also not reproduce any difference between ecj versions
within the range 3.6 - 4.3RC4.


Without evidence that ecj behavior indeed changed this doesn't classify as a 
regression and fixing for Luna might be sufficient.

Am I missing anything?
Comment 4 Martijn Dashorst CLA 2013-06-11 06:37:22 EDT
This is the distribution I downloaded that exhibits the compiler error:

> Eclipse Java EE IDE for Web Developers.
> 
> Version: Kepler Release
> Build id: 20130606-0932
> 
> (c) Copyright Eclipse contributors and others 2005, 2013.  All rights reserved.
> Visit http://www.eclipse.org/webtools

I'll upload the full project as an attachment (perhaps the project settings are different).
Comment 5 Martijn Dashorst CLA 2013-06-11 06:38:11 EDT
Created attachment 232227 [details]
Complete project that shows the behavior from comment #0
Comment 6 Martijn Dashorst CLA 2013-06-11 06:42:25 EDT
Created attachment 232228 [details]
Eclipse configuration

Added my eclipse configuration for 4.3RC3 (JEE download) to give insight into which plugins are installed.
Comment 7 Stephan Herrmann CLA 2013-06-11 07:39:21 EDT
Thanks, the attachment from comment 5 helps to see: not the project settings make
the difference, but whether or not SubSub implements foo :)
Comment 8 Stephan Herrmann CLA 2013-06-11 07:46:43 EDT
With the corrected example I could narrow it down to being a regression caused by commit 4dd974a226271180ff02d909a12722017f80ff3a on behalf of bug 395681. Sigh!

Note, however, that the regression only concerns the compile error, not the behavioral difference to javac. In the end we may have to split the fix into fixing the regression (for 4.3.1) and addressing the behavioral difference (for 4.4 - assuming that indeed ecj is wrong, could also be javac).
Comment 9 Jay Arthanareeswaran CLA 2013-08-21 01:50:50 EDT
(In reply to comment #8)
> Note, however, that the regression only concerns the compile error, not the
> behavioral difference to javac. In the end we may have to split the fix into
> fixing the regression (for 4.3.1) and addressing the behavioral difference
> (for 4.4 - assuming that indeed ecj is wrong, could also be javac).

Stephan, now that we are very close to 4.3.1, as you said, we should perhaps fix only the compiler regression part for 4.3.1 and take a decision on the rest later?
Comment 10 Stephan Herrmann CLA 2013-08-21 16:12:23 EDT
Let me gather some relevant information:

The situation triggering the compiler error looks identical to bug 411811. The relevant code portion here is:

	public static class F1 { }

	public static class F2 extends F1 { }

	public static class F3 extends F2 { }

	public static abstract class Base<T extends F1> {
		public abstract void foo(T bar);
	}

	public static abstract class Sub<T extends F2> extends Base<T> {
		@Override public void foo(F2 bar) { }
	}

	public static class SubSub extends Sub<F3> { }

Here the question is, whether SubSub needs its own method to implement Base.foo(T).
Looking from SubSub that method spells out as "foo(F3)" and SubSub does not declare nor inherit any concrete method with a compatible signature. That's the error we started reporting with Kepler.

In bug 411811 comment 4 I argued at some more detail, why I believe this new behavior to be correct wrt JLS7.

Next I asked about this in the JSR 335 EG, because the new draft spec differs in these points, and I wanted to check if the change was intended, accidental or what:
http://mail.openjdk.java.net/pipermail/lambda-spec-experts/2013-July/000308.html

The last answer in that thread states: "the overriding relation is monotonic with respect to inheritance: you can add new overrides via inheritance, but can't "undo" existing ones."

While this is not part of the spec nor explicitly backed by the spec, we should probably take that as the *intended spec*.


Unfortunately, the notion of override monotonicity doesn't easily translate into an implementation strategy, because method verification always happens for each type in isolation. We start with a flat set of inherited methods, rather than with the result of method verification for the super class.

I'll see if I find an easy solution.
Comment 11 Jay Arthanareeswaran CLA 2013-08-22 06:28:54 EDT
*** Bug 415600 has been marked as a duplicate of this bug. ***
Comment 12 Stephan Herrmann CLA 2013-08-22 16:02:58 EDT
Created attachment 234677 [details]
patch under test
Comment 13 Stephan Herrmann CLA 2013-08-22 16:22:13 EDT
Implementation Notes:
---------------------
Before the patch, the check isSkippableOrOverridden() directly compared signatures using isParameterSubsignature(specific, general). Since both methods have been instantiated from the perspective of the focus type (SubSub in the problem case), we didn't find a match.

Using isMethodSubsignature(specific, general) instead, we now use 'specific.original()' and adjust 'general' accordingly. This reflects the perspective from the declaration of 'specific'. Doing so assures that the original overriding relationship (as seen in Sub) is still recognized.

This change caused regressions first against bug 346029 later bug 195802.
(Cited tests are in MethodVerifyTests).

1) In test346029() we detected too many matches, pulling up the check 'specific.declaringClass.isCompatibleWith(general.declaringClass)' one level solved the regression. Now it looks a bit funny that skip[idx] and isOverridden[idx] are assigned in lock step. Once we are sure this is the right fix, we should check if this can be simplified (outside isSkippableOrOverridden() these arrays are still used independently).

2) Finally test146 (on behalf of bug 195802) reported its original bogus "duplicate" error. Since the original fix can no longer be recognized in the code (due to subsequent restructuring), I freshly special-cased this situation. The situation can be recognized by a pair of methods, where one is the rawified PGMB of the other. I believe this to be a precise description of the case in bug 195802, so directly avoiding the duplication error should be OK.
Comment 14 Stephan Herrmann CLA 2013-08-22 17:09:55 EDT
I have added more tests from duplicates (bug 411811 and bug 415600) - both are green with the patch.

I have released the patch for *master* (commit ce089180323dc4b5ef74d43ea0b4475c385fcd97) and would like to ask everybody to test this patch since it touches a brittle area of the compiler.

If anybody has the time for some extra testing I suggest to use other bugs for inspiration: the duplicates mentioned above and also bug 195802 and bug 346029, which were broken by intermediate versions of the patch. One idea would be to try to construct some nasty tests by using interfaces where existing tests only use classes.

I hope we can get sufficient confidence for back porting in time for SR1 RC2.
Comment 15 Dani Megert CLA 2013-08-23 04:49:57 EDT
+1 for 4.3.1.
Comment 16 Jay Arthanareeswaran CLA 2013-08-26 10:24:52 EDT
(In reply to comment #14)
> I hope we can get sufficient confidence for back porting in time for SR1 RC2.

I did some testing and no issues so far.
Comment 17 Stephan Herrmann CLA 2013-08-27 03:55:43 EDT
Released for 4.3.1 via commit 288e8d99c513639a6f3fdf977634474ff458e3aa
Comment 18 Stephan Herrmann CLA 2013-08-27 04:00:19 EDT
*** Bug 411811 has been marked as a duplicate of this bug. ***
Comment 19 Jay Arthanareeswaran CLA 2013-08-29 06:02:53 EDT
Verified for 4.3.1 with build M20130828-0800
Comment 20 Tomasz Stanczak CLA 2013-09-10 06:56:17 EDT
Founds another case for this still not working. 
Using Version: 4.3.1, Build id: M20130905-0705.
See the attachment: the class Worker3 causes the same error. In v3.7.1 it compiles ok.
Comment 21 Tomasz Stanczak CLA 2013-09-10 06:57:37 EDT
Created attachment 235343 [details]
Another case, where the solution doesn't work yet.
Comment 22 Dani Megert CLA 2013-09-10 06:58:39 EDT
(In reply to Tomasz Stanczak from comment #21)
> Created attachment 235343 [details]
> Another case, where the solution doesn't work yet.

Can you please open a new bug for this. Thanks.
Comment 23 Stephan Herrmann CLA 2013-09-10 10:12:19 EDT
*** Bug 416917 has been marked as a duplicate of this bug. ***