Bug 565063 - JUnit test execution -- org.eclipse.cdt.ui.tests depend on run order
Summary: JUnit test execution -- org.eclipse.cdt.ui.tests depend on run order
Status: NEW
Alias: None
Product: CDT
Classification: Tools
Component: cdt-editor (show other bugs)
Version: 9.11.0   Edit
Hardware: PC All
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact: Jonah Graham CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-08 15:23 EDT by Martin Weber CLA
Modified: 2020-07-11 10:57 EDT (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Weber CLA 2020-07-08 15:23:23 EDT 
Tests in org.eclipse.cdt.ui.tests fail if they are not run in the order imposed by the AutomatedSuite. IMHO, *unit-tests* should be be able to run on their own.

While trying to solve bug 564002 which eliminates the need to code extra AutomatedIntegrationSuite classes, I found that BasicCallHierarchyTest starts to fail.
This seems to be caused by the now different order of test execution. BasicCallHierarchyTest is no longer the first test executed in package org.eclipse.cdt.ui.tests.callhierarchy.

Here is a short comparision:

1) AutomatedIntegrationSuite dictates the order of test execution:
  No failures.
2) The maven-surefire-plugin runs the tests in its default order (filesystem dependent):
  New failures
  Test Result (8 failures / +8)
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testEnumeratorCpp
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testAnonymousEnumeratorCpp_156671
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testAnonymousStructMembersCpp_156671
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testAnonymousUnionMembersCpp_156671
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testStructMembersCpp
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testVariablesCpp
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testFunctionsCpp
    org.eclipse.cdt.ui.tests.callhierarchy.BasicCallHierarchyTest.testUnionMembersCpp
3) The maven-surefire-plugin runs the tests in its alphabetical order:
  New failures:
  Test Result (33 failures / +33)
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testUsingCompletionWithFollowingSemicolon
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCompletionInNamespaceAlias_545360
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testExistingParens_72391
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testForStatement2
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testNoDefaultFunctionParameter
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testQualificationForAccess1
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testQualificationForAccess2
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCunstructorInitializerList_NameContextInput_266586
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testUniformInitializationInConstructorChainInitializer_509185
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testAliasTemplate_455797
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCunstructorInitializerList_MemberInput_266586
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testNoDefaultFunctionArgument
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testFunctionWithNoParameterGuesses_497190
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testBindingsWithoutDeclaration
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCursorPositionForZeroArgFunction_537031
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testDependentScopes_472818d
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testLambdaParameter_481070
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testDefaultFunctionArgument
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testUsingDeclaration_379631
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testBuiltinMacroSegmentMatch
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testReplacementLength_511274
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testNestedAnonymousStructs_206450
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testFunctionWithTypedefToAnonymousType_bug192787
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testNestingClassNameInCompletion_395571
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCompletionInInitializerList_230389
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testAliasTemplateTypeSpecifier_521820
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testCompletionAfterLessThan_229062
    org.eclipse.cdt.ui.tests.text.contentassist2.CompletionTests.testConstructorInitializerList_EmptyInput_266586
    org.eclipse.cdt.ui.tests.text.contentassist2.HelpProposalTests.testInactiveCodeBlock_72809
    org.eclipse.cdt.ui.tests.text.contentassist2.HelpProposalTests.testHelpProposalClobberingTokens_391439
    org.eclipse.cdt.ui.tests.text.contentassist2.ShowCamelCasePreferenceTest.testCamelCaseOn
    org.eclipse.cdt.ui.tests.text.contentassist2.ShowCamelCasePreferenceTest.testDefault
    org.eclipse.cdt.ui.tests.text.contentassist2.ShowCamelCasePreferenceTest.testCamelCaseOff

In 3), callhierarchy.BasicCallHierarchyTest is the first test to run (through the alphabetical order) just like it was defined by callhierarchy.CallHierarchyTestSuite and the test failures are gone. But with alphabetical order, the test in package contentassist2 start to fail, revealing order-dependence of their own.