Bug 302460 - NullPointerException in BcelTypeMunger.mungeNewMethod()
Summary: NullPointerException in BcelTypeMunger.mungeNewMethod()
Status: RESOLVED FIXED
Alias: None
Product: AspectJ
Classification: Tools
Component: Compiler (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 1.6.9M1   Edit
Assignee: aspectj inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-02-10 12:00 EST by Ludger Solbach CLA
Modified: 2010-04-16 13:10 EDT (History)
1 user (show)

See Also:


Attachments
ajcore file of an ant build showing the problem (142.40 KB, text/plain)
2010-02-11 05:37 EST, Ludger Solbach CLA
no flags Details
Remove some of the weakreferences (10.16 KB, application/octet-stream)
2010-02-19 13:29 EST, Andrew Clement CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ludger Solbach CLA 2010-02-10 12:00:23 EST
Build Identifier: 2.0.2

java.lang.NullPointerException
at org.aspectj.weaver.bcel.BcelTypeMunger.mungeNewMethod(BcelTypeMunger.java:909)
at org.aspectj.weaver.bcel.BcelTypeMunger.munge(BcelTypeMunger.java:94)
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:448)
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:103)
at org.aspectj.weaver.bcel.BcelWeaver.weave(BcelWeaver.java:1831)
at org.aspectj.weaver.bcel.BcelWeaver. ...                   ARETURN
  end public String toString()
end public abstract class package.AbstractClassName

Looks like a compiler bug. I believe that I have seen this in an ant build with AspectJ 1.6.8 too. But it happens often in "Project -> Clean" in a complex multi project setup.

Reproducible: Sometimes
Comment 1 Ludger Solbach CLA 2010-02-11 05:37:40 EST
Created attachment 158830 [details]
ajcore file of an ant build showing the problem

I have encountered the problem again in an ant build. Here's the ajcore dump...
Comment 2 Ludger Solbach CLA 2010-02-11 05:43:20 EST
Next try to build the projects via ant with the same build command and without any project changes worked flawless. It really only happens sometimes but more often in ajdt then in iajc.
Comment 3 Andrew Clement CLA 2010-02-11 15:47:01 EST
The exception trace indicates that there was a problem processing the 'top most implementor' of an interface.  The idea is that for an ITD onto an interface, the top most implementor of the interface becomes the host for any state (itd fields) or code (itd methods).  At the point of the NPE we have discovered that what we thought was the top most implementor actually isnt.  Unfortunately there isn't enough information to tell me whether:

- it isn't the top most implementor because it has nothing to do with the target interface for the ITDs

or

- it isn't the top most implementor because someone further up the superclass hierarchy for the type is actually the top most implementor

I don't just want to guard the null because you are seeing intermittent behaviour, so I might add some better diagnostics here that will come through in a dev build shortly.

The intermittent nature can just be due to the order in which the files are processed.  You may be able to affect whether it occurs by turning off pipe line compilation: -Xset:pipelineCompilation=false

the -Xset option can be supplied to ajc or into Ant with (I think) X="-Xset:pipelineCompilation=false"
Comment 4 Ludger Solbach CLA 2010-02-12 09:34:35 EST
Could it be possible that it has to do with unweaving/reweaving of the same ITDs, which happens because the aspects are framework aspects which get applied more than once because already woven classes are in the inpath of a project further downsteam? That happens in our project setup...
Comment 5 Andrew Clement CLA 2010-02-12 11:35:33 EST
I don't think it will be reweaving that is the problem, but that is interesting to know.

The latest dev build includes some debug so it won't NPE now, it will produce error messages instead telling us the types involved which will then enable us to work out what it is doing wrong.  Download it and give it a spin if you can, let me know the outcome: http://eclipse.org/aspectj/downloads.php
Comment 6 Ludger Solbach CLA 2010-02-12 11:49:32 EST
I'm out of office until Tuesday, but then I give it a try.
Comment 7 Ludger Solbach CLA 2010-02-17 07:53:42 EST
I'd like to give the dev version a try today. Is it possible to use this version in AJDT 2.0.2 (in a reversable manner)? There I see the bug more often, so the chance to get an ajcore file from inside AJDT is higher.
Comment 8 Andrew Clement CLA 2010-02-17 13:12:25 EST
It isn't as trivial as it could be because 1.6.9 (the current dev builds) has some radical changes for ITD fields.  However, if I give you rebuilt jars for the plugins, you could try it out.

I've uploaded a jar aj169.jar to here:

http://www.eclipse.org/downloads/download.php?file=/tools/aspectj/dev/aj169.jar

inside are an ajde.jar and an aspectjweaver.jar - these are replacements for the jars within your org.aspectj.ajde and org.aspectj.weaver plugins that you have installed as part of AJDT.  So, shutdown eclipse, go into those two plugin directories under your eclipse install, backup the old jars, replace them with those from aj169.jar then restart eclipse.
Comment 9 Ludger Solbach CLA 2010-02-17 18:51:04 EST
I'll try the replacement jars for AJDT tomorrow. I did about 15-20 full builds with ant/iajc today with the dev version, but have not encounered the bug. Just when you need it, it doesn't show up. :-|
Comment 10 Ludger Solbach CLA 2010-02-18 05:49:25 EST
I've exchanged the jars in AJDT with the ones you provided.

Now I sometimes get errors (in problems view and as gutter markers) for the following ITD methods below. 

Example description from the Problems view:

ITD target de.dsvgruppe.zebra.admin.domain.model.impl.AbstractOrganisation doesn't appear to implement de.dsvgruppe.zebra.admin.domain.historization.HistorizationAspect$Historized why did we consider it the top most implementor? ITD is java.util.List<de.dsvgruppe.common.domain.object.HistorizationEvent> de.dsvgruppe.zebra.admin.domain.historization.HistorizationAspect$Historized.getHistorizationEventList()	HistorizationAspect.aj

Repeated for the other ITDed Methods and other annotated abstract classes.


The type AbstractOrganisation has a @Historizable TYPE annotation with retention RUNTIME and comes from a JAR file which is on the in path of the project containing the HistorizationAspect and the concrete implementation 

Offending aspect:

public aspect HistorizationAspect {
	// Introduce Historized interface on Historizable annotated domain objects
	declare parents : (@de.dsvgruppe.common.annotation.domain.Historizable *) implements Historized;	

	interface Historized {
		List<HistorizationEvent> getHistorizationEventList();
	}

	// Intertype declarations for Historized interface
	private List<HistorizationEvent> Historized.histEventList = new ArrayList<HistorizationEvent>();

	private synchronized void Historized.addHistorizationEvent(HistorizationEvent event) {
		histEventList.add(event);
	}

	public synchronized List<HistorizationEvent> Historized.getHistorizationEventList() {
		return Collections.unmodifiableList(histEventList);
	}
	
	public synchronized void Historized.clearHistorizationEventList() {
		histEventList.clear();
	}

	// NOTE: pointcut and advives omitted
        ...
}
Comment 11 Andrew Clement CLA 2010-02-18 13:26:53 EST
excellent, it fired off the diagnostics perfectly.

So from the look of your aspect the type AbstractOrganisation ought to implement Historized - due to the declare parents and the annotation.

So the problem isn't in computing the top most implementor, it is with remembering the augmented set of interfaces for the affected target type (AbstractOrganisation).

I'm left suspecting that it is the weak reference created in the ReferenceType.getDeclaredInterfaces() method.  Because it is a reference it could be cleared on any random GC - and thus whether it is being used could vary across runs.  However, from looking at the code I can't see the problem.  The reference (which is used to cache the resolved set of interfaces) is cleared any time there is a call to add a new parents (Historized in this case) which would mean on next call to getDeclaredInterfaces() the cached value is recomputed.

I suppose the other option would be the ordering of source file processing - did you try the -Xset:pipelineCompilation=false option I mentioned in comment 3?  It would be nice to eliminate pipelining as a cause.
Comment 12 Ludger Solbach CLA 2010-02-19 05:15:46 EST
I've added -Xset:pipelineCompilation=false in the "Non-standard compiler options" field in the aspectj compiler preferences in eclipse. But the error is still triggered.

In the release notes of 1.6.8 you mentioned the change from Soft- to WeakReferences in 1.6.7. As I haven't encountered the bug before the update to AJDT 2.0.2 I think it is quite possible that the bug is connected with this change.
Comment 13 Andrew Clement CLA 2010-02-19 11:35:04 EST
thanks for checking that out - I'll continue to puzzle through some scenarios that could lead to the cached reference not being cleared when it should be.
Comment 14 Andrew Clement CLA 2010-02-19 13:29:56 EST
Created attachment 159599 [details]
Remove some of the weakreferences

I'm going crazy trying to find a path that will result in the weakreference being the problem.  So... I thought I'd create a patch that contains a version of the class without caching in it.  This will tell me if it is that reference I mentioned.  If you get a chance, can you apply this patch to the aspectjweaver.jar in the org.aspectj.weaver plugin (it will need to be applied over the top of the new jars I gave you).

jar -xvf patch.zip
jar -uvf aspectjweaver.jar org
Comment 15 Ludger Solbach CLA 2010-02-23 03:14:50 EST
ok, I've patched the weaver. Let's see what happens. Do I assume correctly, that I shouldn't experience any errors now, if the problem is related with the WeakReference?
Comment 16 Andrew Clement CLA 2010-02-23 12:35:09 EST
yes, if the problem is that the weakreference was not being cleared at the right time, then you won't see it now.
Comment 17 Ludger Solbach CLA 2010-02-24 12:56:55 EST
The bug is still there.

And on another build (without any code changes) i've got this exception too: 

org.aspectj.weaver.BCException
at org.aspectj.weaver.bcel.BcelRenderer.visit(BcelRenderer.java:205)
at org.aspectj.weaver.ast.Literal.accept(Literal.java:29)
at org.aspectj.weaver.bcel.BcelRenderer.recur(BcelRenderer.java:111)
at org.aspectj.weaver.bcel.BcelRenderer.renderTest(BcelRenderer.java:101)
at org.aspectj.weaver.bcel.BcelAdvice.getTestInstructions(BcelAdvice.java:648)
at org.aspectj.weaver.bcel.BcelAdvice.getAdviceInstructions ... bra.admin.domain.impl.UserImplTest.testUnlock())
  end public void testUnlock()
end public class de.dsvgruppe.zebra.admin.domain.impl.UserImplTest
Comment 18 Andrew Clement CLA 2010-02-24 13:38:12 EST
Thanks for checking that out - I guess it confirms the code in that area was behaving as I expected.  There are other reference caches here and there, guess I'll look at those now.
Comment 19 Andrew Clement CLA 2010-02-24 16:09:51 EST
someone on the mailing list is also reporting a declare parents type problem, interestingly they are also doing what is mentioned here in comment 4 - reweaving.
Comment 20 Andrew Clement CLA 2010-02-24 16:17:20 EST
is there any way you can share the failing project with me?  That would really help me get to the bottom of this quickly.
Comment 21 Ludger Solbach CLA 2010-02-26 09:07:43 EST
I'm afraid that there's no way to share the actual project on various reasons.
I'll try to recreate the failing scenario with dummy projects and share them when I have a bit of time. I'm currently in midst of a release till next week.
Comment 22 Andrew Clement CLA 2010-03-03 14:29:58 EST
fix for 303924 that I just put in is likely to affect this bug too - may fix it.  Fix is in the next dev build.
Comment 23 Ludger Solbach CLA 2010-03-04 11:17:06 EST
could you supply me the replacement jars for AJDT of the latest dev build? Then I'll try them out on our code and test whether it fixes this bug too.
Comment 24 Andrew Clement CLA 2010-03-04 12:26:22 EST
replacement aspectjweaver.jar for plugin org.aspectj.weaver is here:
http://www.eclipse.org/downloads/download.php?file=/tools/aspectj/dev/aspectjweaver-169dev.jar

replacement ajde.jar for plugin org.aspectj.ajde is here:
http://www.eclipse.org/downloads/download.php?file=/tools/aspectj/dev/ajde-169dev.jar

i'm hopeful this will make a difference.
Comment 25 Ludger Solbach CLA 2010-03-05 11:08:50 EST
It looks fine at the first glance. I've done about 20 builds in eclipse and haven't seen the bug. With the old version I ran into this bug at least on every third build. I'll keep on watching and give you further feedback next week.
Comment 26 Andrew Clement CLA 2010-03-05 11:28:52 EST
thanks for testing that out.

The fix was to alter a weak reference cache that was new in 1.6.7.  It wasn't in ReferenceType (where I initially patched) but in BcelObjectType.   The cache wasn't being cleared correctly.  let's give it a few more days and then I'll close this if you haven't seen the issue.
Comment 27 Andrew Clement CLA 2010-03-10 11:46:34 EST
believed fixed!
Comment 28 Ludger Solbach CLA 2010-03-10 12:43:19 EST
I haven't seen the bug since the installation of your replacement jars for AJDT.
It seems to be fixed now.

Thanks Andy!