Bug 473657 - [1.8][compile][inference] Bad type inference applied to raw type arguments when javac compiles code just fine
Summary: [1.8][compile][inference] Bad type inference applied to raw type arguments wh...
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.5   Edit
Hardware: All All
: P4 normal (vote)
Target Milestone: 4.6 M3   Edit
Assignee: Stephan Herrmann CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-07-27 09:02 EDT by Lukas Eder CLA
Modified: 2022-11-01 06:17 EDT (History)
3 users (show)

See Also:
sasikanth.bharadwaj: review+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lukas Eder CLA 2015-07-27 09:02:07 EDT
The following example can be compiled with javac 1.8.0_51 but not with Eclipse 4.5.0:

------------------------------------------------------------
interface I<T> {
}

@SuppressWarnings({"unchecked", "rawtypes"})
abstract class T1<T> implements I<T> {
    public I<T> t(I<? extends Number> l2) {
        return T2.m((I) this, (I) l2);
    }
    public I<T> t(Number l2) {
        return T2.m((I) this, (I) T2.t(l2));
    }
}

abstract class T2 {
    public static <T> I<T> t(T t) {
        return null;
    }
    public static <T extends Number> I<T> m(I<T> l1, I<? extends Number> l2) {
        return null;
    }
    public static <T extends Number> I<T> m(T l1, Number l2) {
        return null;
    }
}
------------------------------------------------------------

Eclipse reports in both T1.t() methods:

> Type mismatch: cannot convert from I<Number> to I<T>

A workaround using an explicit raw type cast can be seen here:

------------------------------------------------------------
    public I<T> t(I<? extends Number> l2) {
        return (I) T2.m((I) this, (I) l2);
    }
    public I<T> t(Number l2) {
        return (I) T2.m((I) this, (I) T2.t(l2));
    }
------------------------------------------------------------

However, this workaround doesn't work well as Eclipse's "remove unnecessary casts" save action will remove the two raw type casts, as they're indeed not needed.
Comment 1 Lukas Eder CLA 2015-07-27 09:05:21 EDT
Another workaround is this:

------------------------------------------------------------
    public I<T> t(I<? extends Number> l2) {
        final I result = T2.m((I) this, (I) l2)
        return result;
    }
    public I<T> t(Number l2) {
        final I result = T2.m((I) this, (I) T2.t(l2));
        return result;
    }
------------------------------------------------------------
Comment 2 Stephan Herrmann CLA 2015-07-27 11:43:07 EDT
Here are my (personal) preferences:

1. avoid raw types.

2. avoid raw types!

3. work on issues with strong indication that ecj is wrong. Deviations wrt javac which involve raw types do not create such strong indication.

4. use explicit type arguments rather than casts, e.g.:
      T2.<I>m(this, l2);
  (disclaimer: not tried in this particular example).
  Rationale: it's the type arguments which are being inferred, hence
  explicitly providing this information is better than inserting casts to
  indirectly influence inference.
  If you specify the type argument no inference is needed.


NB: flagging casts as unnecessary, which were needed to produce the inference result , which then lets us conclude that the cast is unnecessary, is a known issue , for which no trivial solution exists - we may have to re-run inference without the cast to see if it makes a difference.
Comment 3 Lukas Eder CLA 2015-07-27 11:51:56 EDT
I obviously concur with 1) and 2), but sometimes, those raw types can just be very useful ;)

I understand that raw types are very hard to get "right" when implementing a Java compiler, but this particular case is a regression in a recent Mars milestone (probably between M5 and RC1, I think). So I thought I should still report it.
Comment 4 Stephan Herrmann CLA 2015-07-27 16:47:03 EDT
(In reply to Lukas Eder from comment #3)
> I obviously concur with 1) and 2), but sometimes, those raw types can just
> be very useful ;)

almost as useful as (void *)
;P

> I understand that raw types are very hard to get "right" when implementing a
> Java compiler,

*maybe* we already got it right?

> but this particular case is a regression in a recent Mars
> milestone (probably between M5 and RC1, I think).

That's not what I see. I see
- at -1.8 ecj consistently raises two errors (ever since P20140317-1600)
- at -1.7 ecj consistently accepts the snippet

This kind of difference shouldn't necessarily surprise.
Comment 5 Lukas Eder CLA 2015-07-28 02:26:02 EDT
> almost as useful as (void *)
> ;P

I do miss pointer magic in Java ;)

> *maybe* we already got it right?

Hard to say. Want to do the exercise of proving it against the JLS? :) But then, how would you explain that this workaround still works?

------------------------------------------------------------
    public I<T> t(I<? extends Number> l2) {
        final I result = T2.m((I) this, (I) l2)
        return result;
    }
------------------------------------------------------------

> That's not what I see. I see
> - at -1.8 ecj consistently raises two errors (ever since P20140317-1600)
> - at -1.7 ecj consistently accepts the snippet

Hmm, you might be right. In addition to upgrading Eclipse, I had also upgraded the Java language version - I must have forgotten about that.
Comment 6 Timo Kinnunen CLA 2015-07-28 16:05:42 EDT
It looks like javac infers the return type of the method invocation to be the raw type I when passed in arguments are raw types. This may be motivated by the almost last bit of §18.5.2 after bounds set 4 has been generated:

Finally, if B4 does not contain the bound false, the inference variables in B4 are resolved. 

 If resolution succeeds with instantiations T1, ..., Tp for inference variables α1, ..., αp, let θ' be the substitution [P1:=T1, ..., Pp:=Tp]. Then: 

◦ If unchecked conversion was necessary for the method to be applicable during constraint set reduction in §18.5.1, then the parameter types of the invocation type of m are obtained by applying θ' to the parameter types of m's type, and the return type and thrown types of the invocation type of m are given by the erasure of the return type and thrown types of m's type. 

So if unchecked conversion had to be used earlier then the method return type just gets erased unceremoniously at the end.
Comment 7 Stephan Herrmann CLA 2015-07-28 17:26:02 EDT
(In reply to Timo Kinnunen from comment #6)
> ◦ If unchecked conversion was necessary for the method to be applicable
> during constraint set reduction in §18.5.1, then the parameter types of the
> invocation type of m are obtained by applying θ' to the parameter types of
> m's type, and the return type and thrown types of the invocation type of m
> are given by the erasure of the return type and thrown types of m's type. 

Cool, thanks for digging this out.

This section was an early Xmas gift less than three months before Java 8 GA (first appeared in JLS 335 spec v0.9.0, 2013-12-21). Should have been adopted via bug 424286 but I must have been distracted by s.t. ;P

I'd love to make a quick experiment to see what breaks when we insert this erasure. Unfortunately, we seem to have a technical difficulty as we don't have the infrastructure to manufacture a ParameterizedGenericMethodBinding that applies special substitution rules for return and thrown exceptions which are different from those used for parameters.

At this point I have the uneasy feeling of having been in this very same position some 18 months ago. But after much digging I just can't find any record of such prior discussion :(
Comment 8 Eclipse Genie CLA 2015-08-29 12:36:53 EDT
New Gerrit change created: https://git.eclipse.org/r/54835
Comment 9 Eclipse Genie CLA 2015-08-30 02:17:04 EDT
New Gerrit change created: https://git.eclipse.org/r/54843
Comment 10 Stephan Herrmann CLA 2015-08-30 03:35:59 EDT
(In reply to Eclipse Genie from comment #9)
> New Gerrit change created: https://git.eclipse.org/r/54843

Sasi, this patch may still require cleanup / documentation, but perhaps you can already do a quick review at your convenience?
Comment 11 Stephan Herrmann CLA 2015-09-12 12:27:16 EDT
(In reply to Stephan Herrmann from comment #10)
> (In reply to Eclipse Genie from comment #9)
> > New Gerrit change created: https://git.eclipse.org/r/54843
> 
> Sasi, this patch may still require cleanup / documentation, but perhaps you
> can already do a quick review at your convenience?

I forgot to add the test, and it turns out I wasn't done yet.

After adding the forgotten raw conversion, the test was in conflict with a tweak in ConstraintExpressionFormula.inferPolyInvocationType() bearing the following comment:
	// continuing at true is not spec'd but needed for javac-compatibility,
	// see org.eclipse.jdt.core.tests.compiler.regression.GenericsRegressionTest_1_8.testBug428198()
	// and org.eclipse.jdt.core.tests.compiler.regression.GenericsRegressionTest_1_8.testBug428264()
Due to continuing where according to the spec we shouldn't we actually had a "T#0 = T" type bound in the mix which conflicted with "T#0 <: Number" (from <T>'s type bound).
Returning from this methods as spec'ed avoids generating the bogus type bound here.
	
I was happy to see, that the new fix actually obsoletes the previous hack, i.e., the two bugs mentioned in the comment no longer fail with the proper "return" in place.

Circling back to bug 428198 comment 5 and for maximum transparency I also dug out the message by Dan, which should have told me all this already: http://mail.openjdk.java.net/pipermail/lambda-spec-experts/2014-March/000508.html
But then, 4 days before Java 8 GA I was perhaps more concerned about travel preparations for ECNA than about delicate bug fixes :)
Comment 12 Eclipse Genie CLA 2015-09-12 12:29:19 EDT
New Gerrit change created: https://git.eclipse.org/r/55801
Comment 13 Stephan Herrmann CLA 2015-09-12 17:22:59 EDT
(In reply to comment #12)
> New Gerrit change created: https://git.eclipse.org/r/55801

Hudson still wasn't satisfied.

Some tests showed slightly different error messages, at a quick glance, these differences are not very relevant -> tests adjusted.

GRT.testBug454644() (was wrongly named testBug453253()) went back to *not* reporting an error, which surprises me, but now ecj and javac agree on this test -> test adjusted.

Other regressions required me to tweak the correction: apparently type variables must not be erased by the rule in 18.5.2. See PGMB.getErasure18_5_2() and the remaining regressions caused when unconditionally returning the raw type. Perhaps we should substitute-then-erase? The spec doesn't seem to help us here, it just says "erasure".
Comment 14 Stephan Herrmann CLA 2015-09-12 17:31:15 EDT
(In reply to comment #13)
> See PGMB.getErasure18_5_2() and the remaining regressions caused when unconditionally returning the raw type.

These are (in GTT at 1.8):
- test1012
- test1013
- test1436
- test1437
- test1438
- test1439
- test1445
plus less relevant changes in error messages:
- test1325
- test1434
Comment 15 Stephan Herrmann CLA 2015-09-12 18:03:32 EDT
(In reply to comment #13)
> Other regressions required me to tweak the correction: apparently type variables
> must not be erased by the rule in 18.5.2. See PGMB.getErasure18_5_2() and the
> remaining regressions caused when unconditionally returning the raw type.
> Perhaps we should substitute-then-erase? The spec doesn't seem to help us here,
> it just says "erasure".

Indeed JavaAllJava8Tests is also content with this one-liner:

    		return env.convertToRawType(Scope.substitute(this, type), true);

Slightly more work, but this would be a more-likely interpretation of JLS, wouldn't it?
Comment 16 Stephan Herrmann CLA 2015-09-13 09:17:40 EDT
To reduce our deviation from JLS I did:

- switch to substitute-then-erase, as to apply erasure *always*

- reduce this tweak to the following situations (given inferredWithUncheckConversion==true):
 - return type during simulation of Bug JDK 8026527
 - all(!) thrown types 
 
 Since the tweaking of thrown types still isn't kosher (but required e.g. for GTT.test1436) I also asked Dan Smith for clarification.
 
 If Hudson agrees, This would be my solution for M2, pending further updates upon Dan's answer.
Comment 17 Stephan Herrmann CLA 2015-09-13 10:22:02 EDT
(In reply to Stephan Herrmann from comment #16)
>  If Hudson agrees, This would be my solution for M2, pending further updates
> upon Dan's answer.

Patch Set 3 got +1 via https://hudson.eclipse.org/platform/job/eclipse.jdt.core-Gerrit/534/
Comment 18 Stephan Herrmann CLA 2015-09-15 08:47:35 EDT
Thanks, Sasi, for +1.

Since M2 is basically done, I defer releasing to M3 nevertheless. Maybe we even have an answer from Dan by then.
Comment 19 Stephan Herrmann CLA 2015-09-20 09:28:25 EDT
Dan's answer confirms that javac has a bug: https://bugs.openjdk.java.net/browse/JDK-8135087

Substitution should *not* apply here.

To accommodate that hint, I prepared a new patch where the un-spec'ed substitution is applied *only* in the context of SIMULATE_BUG_JDK_8026527.

With this patch the following tests in GTT are changed once more (ignoring simple changes in error message): test1436, test1437, test1438, test1439 and test1445.

- We are introducing a new difference between 1.7 and 1.8 modes in test1436, test1437, test1438 (new error reported).

- We are removing one difference between 1.7 and 1.8 modes in test1439, test1445 (original error retained).

I haven't compared all of those against javac, but I believe any slight deviations to be tolerable given that:
- our new behavior is approved by Dan
- a javac fix is scheduled (only for 1.9, though)
- the situation is a corner case involving all of
-- a throws clause using a type variable
-- a method invocation using unchecked conversions
-- the call site depending (for its catch/throw clauses) on a precisely inferred type variable, where unchecked conversion in fact prohibits such inference result.
Comment 21 Stephan Herrmann CLA 2015-09-20 15:09:44 EDT
(In reply to Eclipse Genie from comment #20)
> Gerrit change https://git.eclipse.org/r/55801 was merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=4e6bb96b0e120539a45cc4860f508bf12e097d37

Resolved for 4.6 M3
Comment 22 Manoj N Palat CLA 2015-10-27 02:59:21 EDT
Verified for Eclipse Neon 4.6M3 Build id: I20151026-2000
Comment 23 Eclipse Genie CLA 2016-04-19 13:31:42 EDT
New Gerrit change created: https://git.eclipse.org/r/70986
Comment 24 Stephan Herrmann CLA 2016-04-19 13:37:38 EDT
(In reply to Eclipse Genie from comment #23)
> New Gerrit change created: https://git.eclipse.org/r/70986

Wrong commit message. Not for this bug but for follow-up bug 491934
Comment 26 Stephan Herrmann CLA 2022-11-01 06:17:33 EDT
For posterity: the story from this bug is continued in https://github.com/eclipse-jdt/eclipse.jdt.core/pull/477

in particular these test changes can essentially be reverted after more tweaks in the compiler:

(In reply to Stephan Herrmann from comment #19)
> With this patch the following tests in GTT are changed once more (ignoring
> simple changes in error message): test1436, test1437, test1438, test1439 and
> test1445.
> 
> - We are introducing a new difference between 1.7 and 1.8 modes in test1436,
> test1437, test1438 (new error reported).
> 
> - We are removing one difference between 1.7 and 1.8 modes in test1439,
> test1445 (original error retained).