Hi, Team,
The Papyrus Diagram Tests Generation Framework, which has been in development for some time and changed hands at least once, has now been integrated into the master (Neon) branch. This framework comprises several components:
- a model transformation pipeline that generates JUnit test suites for the various Papyrus UML diagrams, from several inputs:
- a UML model of the abstract test cases (kinds of tests) that the framework generates. This corresponds to abstract JUnit test classes in the common UML diagram test plug-in and is relatively static, changing only with the transformation rules, codegen templates
- the GMFGen model describing the particular diagram
- an optional enumeration of edit-parts for which to generate tests of each kind supported by the framework, in case any of the test suites that we want to generate is vastly different from what the framework would infer from the GMFGen
- an optional specification of test exceptions for test cases that either shouldn’t be generated at all or at least suppressed via the annotations for failing/interactive/invalid tests. These exceptions may be provided as models or in-line in the transformation workflow in a simple Xtend DSL
- an Xtend program in each diagram test plug-in project that configures and runs the workflow described above to generate particular diagram’s test suite
- a library of abstract test classes in the org.eclipse.papyrus.uml.diagram.common.tests plug-in that the generated tests subclass and call into
- a project nature and builder, installed via the Papyrus Developer Tools feature, that generates the diagram test suites in the developer workspace. Thus
- the generated test code is not managed in the git repository. Any customization to tests must be implemented by custom transformation rules. The Class Diagram and Profile Diagram tests provide examples of customization of the GMFGen-to-UTP m2m transformation phase and the State Machine Diagram Tests provide an example of customization of the UTP-to-Java m2t phase
- your workspace will not incur the overhead of generating the test code for all of the UML diagrams if any one of the following conditions holds:
- you do not have the Papyrus Developer Tools installed in your workbench (they are provided by the Papyrus/Developer/Developer Tools project in the Papyrus Oomph setup model)
- you do not have the org.eclipse.papyrus.tests.framework project in your workspace
- the org.eclipse.papyrus.tests.framework project is in your workspace but is closed
- if the test generation builder is enabled in your workspace per the above conditions, it is only run on a clean build. Incremental builds as you edit code will not trigger test generation
- an @GeneratedTest annotation applied to the generated test suite classes that excludes the generated tests (if you happen to have generated them) from the standard test execution. They take a very long time to run!
One noteworthy addition to the Papyrus core testing library that supports the generated tests is a custom suite runner: ConfigurationSuite is like the standard Suite runner provided by JUnit, except that
- it will entirely exclude a test class if none of that class’s contained methods or sub-suites match the test configuration requested for execution via the testConfig command-line option. This should help to solve the problem of @BeforeClass and @AfterClass steps taking up time in the execution of suites in which all of the tests are skipped, if we replace usage of Suite with ConfigurationSuite generally
- in addition to the @SuiteClasses annotation, a custom @DynamicClasses annotation includes sub-suites by name, not by class reference. This allows to build up a suite of classes that may or may not exist (such as generated tests). Instead of a class loading error from the JVM’s failure to parse an annotation that references a non-existent class, the ConfigurationSuite simply ignores dynamic references that are unresolved
A new test job is provisioned on the Papyrus HIPP instance that runs the entire suite of generated tests every Sunday morning:
This job has yet to produce a result: so far, it has run into pop-up dialogs that cause it to hang, and it has a very long time-out configured because it is expected to be rather long running. I’m still working on getting a useful first test result.
Currently, most of the Papyrus diagram test plug-ins have been configured to generate as nearly as possible the test cases that I think should pass, based on my limited knowledge of these diagrams. We will have to depend on experts in each diagram to analyze the initial test execution results to further trim the test cases and/or raise bugs to fix problems that the tests are uncovering. Diagram test plug-ins that still need to finish the initial test specification process are:
- Sequence Diagram: oep.uml.diagram.sequence.tests
- Timing Diagram: oep.uml.diagram.timing.tests
- Interaction Overview Diagram: oep.uml.diagram.interactionoverview.tests — in this case, nothing has been done at all to implement test generation (the development of the framework predates this diagram)
So, all of this is currently on the master branch for the Neon release. Do we want to do the same on the Mars maintenance branch? All of the work is basically done; it would remain only to submit the Gerrit review that is pending on that branch and set up a test job in Hudson.
This work is tracked in Bugzilla and Gerrit under the following references:
Cheers,
Christian
