Community
Participate
Working Groups
From Thomas Hallgren (BEA): "When investigating some of the failures that JRockit encounters I came across this scenario in the 'update' test org.eclipse.update.tests.regularInstall.TestInstall. Using JRockit, the tests run in the following order: testFileSiteWithoutSiteXML testInstall testHTTPSite testFileSite using the Sun JDK, the order is: testFileSite testHTTPSite testInstall testFileSiteWithoutSiteXML Using the JRockit case, the last test will fail. It consults the org.eclipse.update.internal.core.InstallRegistry.justInstalledPlugins HashMap, concludes that the plugin it is about to install is installed by another feature already, and skips it. Unfortunately, the parent folder of the installed plugin was removed without a corresponding call to InstallRegistry.cleanup() before that. I'd consider this a bug in the test framework. The order by which the tests are run is determined by the JUnit TestSuite class: Method[] methods= superClass.getDeclaredMethods(); for (int i= 0; i < methods.length; i++) { addTestMethod(methods[i], names, theClass); } The documentation on the Class.getDeclaredMethod() says "The elements in the array returned are not sorted and are not in any particular order"."
more.. "When running the org.eclipse.ui.workbench.texteditor.tests.FindReplaceDialogTest, JRockit will use the following order of execution: testDisableWholeWordIfNotWord testDisableWholeWordIfRegEx testInitialButtonState The testInitialButtonState test fail since the isWholeWordSetting is set to true by the preceeding test. A Sun JDK will execute the tests in the following order: testInitialButtonState testDisableWholeWordIfRegEx testDisableWholeWordIfNotWord" ... "I知 beginning to suspect that a vast majority of the errors that we see when running the Eclipse tests with JRockit are contributed to the fact that they depend on a certain order or execution. I consider that a fairly serious flaw in the test framework and something that needs to be fixed in order for me to proceed. Perhaps you could create a patched version of the JUnit TestSuite class that will add the test methods in reverse order. Something like: Method[] methods= superClass.getDeclaredMethods(); int i = methods.length; while(--i >= 0) { addTestMethod(methods[i], names, theClass); } Such a patch will reveal a lot of your issues that are related to order of execution."
I would like to raise the priority of this bug to P1. We are trying to get JRockit certified for Eclipse and this bug effectively blocks that effort completely.
I guess you would have to file that against junit.org where that offending code is located. What I think the answer will be is that a well written test should never depend on the execution order. There are test frameworks that on propose run tests in random order. We are, for example, offering to run only failed tests. So I suggest to move the bug back to the writers of the failing tests and have them make sure that their tests can run in any order. They can also rewrite their tests to not use new TestSuite(Test.class) but suite= new TestSuite(); suite.addTest(new TextCase("test1")): suite.addTest(new TextCase("test2")):
Re-assigning to platform-releng until we have a list of all components that need to revisit their tests.
I've entered bug 142544 and bug 142542 for the scenarios mentioned here.
(In reply to comment #3) > I guess you would have to file that against junit.org where that offending code > is located. > > What I think the answer will be is that a well written test should never depend > on the execution order. > The offending code is definitely *not* in junit.org. I included that code just to show how the order of execution is determined (or rather, not determined). The fact that junit tests run in random order is well known and not a bug. The errors are solely located in the Eclipse test framework itself. The patch that I suggest should be considered temporary. It's just one way to conveniently provoke the errors.
Branko, please take a look at this - seems to be that some Update JUnit test cases are sequential - must be run in a certain order.
Hi Sonia, You seem to be right about this. JUnit tests should be self contianed. I encounter similar issues before with location (drive) of the test. I will investigate more in update but at that time it looked more like tear down of the tests not happen properly or timely. Since, I new how to go around it and both ibm and hotspot VM behaving the same and it happening only on my dev machines I did not take it as priority. But testFileSiteWithoutSiteXML was really main offender and I will now look into it. Once i have resolution I will post my findings. Thomas: Can you try disabling testFileSiteWithoutSiteXML and running update junit tests to see if this is really the offending party.
The tests all run without errors when I run them independently. The whole suite is error free if I force the order to be the same as when using a Sun JVM. Please try my suggested temporary patch for the TestSuite. It will immediately reveal *all* problems of this nature.
Thomas, I think I fixed update junit tests. I can not be 100% sure since i do not have jrockit. Can you obtain them from eclipse CVS and run them?
You can find JRockit downloads here http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/jrockit if you'd care to try it out yourself. An alternative to using JRockit is to install my suggested patch. My quest is to certify JRockit on Eclipse. In order to do so, it must pass all test in the Eclipse test framework. The update junit tests is just one example of tests that fail due to order of execution. You will see that there are a large number of tests in various places (not just in the update and texteditor tests) that fail consistently due to the ordering issue when my suggested temporary patch is used. It seems to me the easiest way to proceed would be to fix them all in one go. I will be happy to verify your CVS version with JRockit as soon as the ordering issue has been addressed everywhere. In order to do so, I'll need instructions on exactly what to download and how to rebuild the test framework from scratch.
Thomas, I am only responsible for update, other junit tests will have to be fixed by other people. I did not apply your patch, however I changed the order of method to get to your order of execution and I after this fix I had no problems. That said I am not sure I can install JRockit due to licensing issues, so I will wait for you to run it.
Sorry, I didn't mean you should fix all the tests. I'm merely suggesting that "someone" could use my patch to reproduce all errors caused by order of execution. Then that "someone" could coordinate the effort to fix all those issues. Who that "someone" is must be decided by people responsible for the Eclipse testing in general. I'm just the bystander who discovered this fairly serious and fairly common flaw in the Eclipse test framework and it's not very efficient for me to try out CVS releases with individual fixes. Regarding JRockit licensing. What issues do you see that would prevent you from installing?
"You will see that there are a large number of tests in various places (not just in the update and texteditor tests) that fail consistently due to the ordering issue when my suggested temporary patch is used." If you have already identified which components are failing with ordering issues, please attach results here and we can open bug reports for those.
Perhaps you could try my suggested patch and run the tests yourself? That way you can do far more then just open individual bug reports. Installing the patch takes a couple of minutes and the resulting setup will help you find, analyze, and also debug all the errors. I don't see how a bunch of results with mixed in failures caused by other aspects of using JRockit would help in that process.
Each test plug-in is contributed by the various components and they are solely responsible for debugging and analyzing their tests. Given the resource constraints there are in releng land, we can certainly help direct issues to the right component owners, but cannot go beyond that until at least July :(
Created attachment 42926 [details] Result output from runtests all Here are the test results when my patch is applied to the junit.framework.TestSuite class. The tests are all run using Sun JDK 1.5.0_07 on a Windows XP workstation. I discovered that the Eclipse that I used (3.2RC6) comes with both junit-3.8.1 and junit4-4.1.0 so I created a patched jar for both. Both the jars and the modified source files are attached. I performed some surgery and replaced the junit.jar in both the eclipse-junit-tests-I20060526-0010.zip and in the eclipse-SDK-I20060526-0010-win32.zip. In the latter I also replaced the junit4.1.jar. That concludes what needs to be done in order to provoke the order dependency errors in almost all the tests. The jdt.core tests however, required some extra attention. They don't use the standard junit TestSuite class. So I did the following: I pached the constructor of the class org.eclipse.jdt.core.tests.junit.extension.PerformanceTestSuite. Same patch as before, simply reverse the order by which the addTestMethod is called. After some tests, it became apparent that this change was not enough. I also had to patch the buildTestsList method of the org.eclipse.jdt.core.tests.junit.extension.TestCase so that it returned the names in reverse order. The patched sources for the PerformanceTestSuite and TestCase are also attached.
Created attachment 42927 [details] Patched version of junit.framework.TestSuite for junit-4.1
Created attachment 42928 [details] Patched version of junit.framework.TestSuite for junit-3.8.1
Created attachment 42929 [details] Patched version of org.eclipse.jdt.core.tests.junit.extension.TestCase
Created attachment 42930 [details] Patched version of org.eclipse.jdt.core.tests.junit.extension.PerformanceTestSuite
Created attachment 42931 [details] Precompiled junit.jar with patch applied
Created attachment 42932 [details] Precompiled junit.jar-4.1 with patch applied
One thing I forgot to mention. The cvs test results shouldn't count. They are caused by an incorrect cvs setup so please disregard them. All other failures and errors (somewhere between 150 and 200) are caused by the change in order of execution. Also, please note. JRockit is *not* needed. This is all done using Sun JDK with my patches installed. Sonia, when you distribute the bugs to their respective owners, would you be so kind and add me (thomas@tada.se) as a CC?
Entered a number of blocking bugs based on results with the exception of the org.eclipse.jdt.debug.tests - there are failures here that appear to be related to a missing src.jar/src.zip in the vm. Darin, could you comment on the JDT Debug test results please and whether there are failures related to an reverse ordering of test method execution?
> ... with the exception of the > org.eclipse.jdt.debug.tests - there are failures here that appear to be related > to a missing src.jar/src.zip in the vm. > The VM in question is a standard Sun JDK 1.5.0_07 installation. It does indeed contain a src.zip so that cannot be the problem.
FYI - when upgrading to JRE 1.5.0_08, there seems to be changes in some hash map implementations which are breaking some of the JDT/Core tests. We are going to address these for 3.2.1; pls advise if these changes are required for JRockit/3.2.0
comment #27 follow-up: I've opened specific bug 146015 for JDT/Core tests remaining problems while running other VMs and put it as prerequesite of this bug...
JRockit will use the same rt.jar as Sun does so I would expect the same problems with hash map changes there but as long as the errors occur on both platforms I doubt it will have a negative impact on JRockit certification. No extra rush needed on our behalf.
from comment 29: HashMap keys order specific issue for JDT/Core tests has been splitted in separated bug 146215 as it does not sound to be a prerequesite for JRockit certification and so does not block current bug...
Created attachment 45836 [details] Program that will merge the managementapi.jar into the rt.jar in a JRockit JRE This is a java program that will merge the managementapi.jar into the rt.jar in a JRockit JVM. The merge is necessary since some tests starts a new JVM with -Xbootclasspath set explicitly to rt.jar. JRockit requires the managementapi classes when debugging.
Can this bug be closed, does it still apply?
I think this bug can be closed. If I recall correctly, the issues with JRockit later became issues with the Sun JVM as well and the problems were fixed in bugs 146015 and 146215. So I'm closing this as fixed.
