Bug 380912 - Unstable test results in 4.2 Hudson build (was: CodeCompletionTest, SemanticHighlightingTest and MarkOccurrenceTest fail in 4.2 build)
Summary: Unstable test results in 4.2 Hudson build (was: CodeCompletionTest, SemanticH...
Status: RESOLVED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Text (show other bugs)
Version: 3.8   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 4.3 M5   Edit
Assignee: JDT-Text-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords: test
Depends on:
Blocks: 381873
  Show dependency tree
 
Reported: 2012-05-29 10:48 EDT by Deepak Azad CLA
Modified: 2013-01-28 05:11 EST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Deepak Azad CLA 2012-05-29 10:48:45 EDT
http://download.eclipse.org/eclipse/downloads/drops4/S-4.2M7-201205031800/testresults/html/org.eclipse.jdt.text.tests_win32.win32.x86_7.0.html

http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC1-201205182145/testresults/html/org.eclipse.jdt.text.tests_win32.win32.x86_7.0.html

http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC2-201205242100/testresults/html/org.eclipse.jdt.text.tests_win32.win32.x86_7.0.html

The failures are in 
org.eclipse.jdt.text.tests.contentassist.CodeCompletionTest
org.eclipse.jdt.text.tests.SemanticHighlightingTest
org.eclipse.jdt.text.tests.MarkOccurrenceTest

The tests pass in the 3.8 builds
http://download.eclipse.org/eclipse/downloads/drops/I20120523-2100/testResults.php
http://download.eclipse.org/eclipse/downloads/drops/I20120527-2100/testResults.php

The failures look a bit mysterious to me, especially the failures in CodeCompletionTest. I have not been able to reproduce the failures on my machine so far.
Comment 1 Deepak Azad CLA 2012-05-29 11:05:56 EDT
There is an exception in the console log (http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC2-201205242100/testresults/consolelogs/win7consolelog.txt)
-----------------------------------------------------------------------------
 [echo] Running org.eclipse.jdt.text.tests.JdtTextTestSuite. Result file: c:\hb\workspace\JUnit-win2\workarea\I20120524-2100\eclipse-testing/results/win32.win32.x86_7.0/org.eclipse.jdt.text.tests.JdtTextTestSuite.xml.
     [java] Exception in thread "Thread-388" java.util.ConcurrentModificationException
     [java] 	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:806)
     [java] 	at java.util.HashMap$KeyIterator.next(HashMap.java:841)
     [java] 	at org.eclipse.jface.text.source.AnnotationModel.getAnnotationIterator(AnnotationModel.java:767)
     [java] 	at org.eclipse.jface.text.source.AnnotationModel.getAnnotationIterator(AnnotationModel.java:691)
     [java] 	at org.eclipse.jface.text.source.projection.ProjectionSummary.removeSummaries(ProjectionSummary.java:173)
     [java] 	at org.eclipse.jface.text.source.projection.ProjectionSummary.internalUpdateSummaries(ProjectionSummary.java:152)
     [java] 	at org.eclipse.jface.text.source.projection.ProjectionSummary.access$3(ProjectionSummary.java:147)
     [java] 	at org.eclipse.jface.text.source.projection.ProjectionSummary$Summarizer.run(ProjectionSummary.java:72)
-----------------------------------------------------------------------------

but that also happens in 3.8 build - http://download.eclipse.org/eclipse/downloads/drops/I20120527-2100/testresults/consolelogs/win7consolelog.txt
Comment 2 Deepak Azad CLA 2012-05-29 11:36:36 EDT
I am clueless now..

- I cannot reproduce this by not having focus in the target workbench
- Tried with different JDKs - 5,6,7
- I do not see any obvious differences in Hudson jobs for 3.8 and 4.2 here - https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/
Comment 3 Dani Megert CLA 2012-05-30 02:27:30 EDT
(In reply to comment #1)
> There is an exception in the console log

So, you can't say that this is related to the failure, right? Please try to trigger the failure on your machine.
Comment 4 Dani Megert CLA 2012-05-30 06:22:54 EDT
> - I cannot reproduce this by not having focus in the target workbench
It depends on the timing. In the IDE you can easily see that the occurrence marks go away when you remove the focus from the workbench window.
Comment 5 Deepak Azad CLA 2012-05-30 06:50:35 EDT
(In reply to comment #4)
> > - I cannot reproduce this by not having focus in the target workbench
> It depends on the timing. In the IDE you can easily see that the occurrence
> marks go away when you remove the focus from the workbench window.

But this behavior is same in 4.2 and 3.8. 

Furthermore, the failure in CodeCompletionTest is more puzzling because there instead of real parameter names we get arg0, arg1 etc even though everything is contained in 1 java file.
Comment 6 Dani Megert CLA 2012-05-30 06:54:41 EDT
> Furthermore, the failure in CodeCompletionTest is more puzzling because there
> instead of real parameter names we get arg0, arg1 etc even though everything > is
> contained in 1 java file.

You mean we get these for source? That would indeed be very strange.
Comment 7 Deepak Azad CLA 2012-05-30 06:56:16 EDT
(In reply to comment #6)
> You mean we get these for source? That would indeed be very strange.
Exactly! 

Which is why I said... :-)
(In reply to comment #2)
> I am clueless now..
Comment 8 Markus Keller CLA 2012-05-30 08:42:17 EDT
A common pattern of these failures is that they need an up-to-date AST from ASTProvider or from SelectionListenerWithASTManager.

If e.g. OverrideCompletionProposal#getRecoveredAST(IDocument, int, Document) returns a wrong AST (e.g. one for the previous editor), then StubUtility#suggestArgumentNames(IJavaProject, IMethodBinding) will fall into "catch (JavaModelException e)" and return "arg0" et al.

However, I can also not reproduce these failures locally.
Comment 9 Deepak Azad CLA 2012-05-30 08:48:12 EDT
(In reply to comment #8)
> A common pattern of these failures is that they need an up-to-date AST from
> ASTProvider or from SelectionListenerWithASTManager.

Interestingly ASTProviderTest also fails

http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC2-201205242100/testresults/html/org.eclipse.jdt.ui.tests_win32.win32.x86_7.0.html

http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC1-201205182145/testresults/html/org.eclipse.jdt.ui.tests_win32.win32.x86_7.0.html
Comment 11 Deepak Azad CLA 2012-06-07 00:42:08 EDT
(In reply to comment #10)
> And now a different set of tests fail - those in the LeakTestSuite
> 
> http://download.eclipse.org/eclipse/downloads/drops4/I20120601-1900/testresults/html/org.eclipse.jdt.ui.tests_win32.win32.x86_7.0.html
> 
> http://download.eclipse.org/eclipse/downloads/drops4/S-4.2RC3-201205311500/testresults/html/org.eclipse.jdt.ui.tests_win32.win32.x86_7.0.html

Opened bug 381942 for these failures.

CodeCompletionTest, SemanticHighlightingTest and MarkOccurrenceTest now pass in the build. Keeping this bug open to make sure that the test results are stable and reliable in the 4.2 build.
Comment 12 Deepak Azad CLA 2012-07-25 06:01:19 EDT
Things are a bit more stable now i.e. the almost the same tests fail, there are only one or two variations.
Comment 13 Dani Megert CLA 2012-11-01 05:00:06 EDT
CodeCompletionTest failed again on Mac.
Comment 14 Dani Megert CLA 2013-01-28 05:11:06 EST
Marking as fixed: the tests were quite stable in the past weeks.