Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] New DSF test structure

I have suites that run all version and all remove versions (AllSuites and AllRemoteSuites) so you don't have to do that, if you find it useful I can create a suite then runs both of these

On Fri, Mar 18, 2016 at 7:36 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:
In case anyone else finds this useful this is the -D for all versions of GDB that had individual tests before the refactor and all the versions buildable with the dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/scripts/download-build-gdb.sh script.

-Dcdt.tests.dsf.gdb.versions=gdb.6.6,gdb.6.7,gdb.6.8,gdb.7.0,gdb.7.1,gdb.7.2,gdb.7.3,gdb.7.4,gdb.7.5,gdb.7.6,gdb.7.7,gdb.7.8,gdb.7.9,gdb.7.10,gdb.7.11,gdbserver.6.6,gdbserver.6.7,gdbserver.6.8,gdbserver.7.0,gdbserver.7.1,gdbserver.7.2,gdbserver.7.3,gdbserver.7.4,gdbserver.7.5,gdbserver.7.6,gdbserver.7.7,gdbserver.7.8,gdbserver.7.9,gdbserver.7.10,gdbserver.7.11

It should be noted that the oldest of those are not supported anymore. see https://wiki.eclipse.org/CDT/User/NewIn86#Discontinued_support_for_some_older_GDB_versions.

For the command line of what we support (>= 7.1)

-Dcdt.tests.dsf.gdb.versions=gdb.7.1,gdb.7.2,gdb.7.3,gdb.7.4,gdb.7.5,gdb.7.6,gdb.7.7,gdb.7.8,gdb.7.9,gdb.7.10,gdb.7.11,gdbserver.7.1,gdbserver.7.2,gdbserver.7.3,gdbserver.7.4,gdbserver.7.5,gdbserver.7.6,gdbserver.7.7,gdbserver.7.8,gdbserver.7.9,gdbserver.7.10,gdbserver.7.11

Jonah

~~~
Jonah Graham
Kichwa Coders Ltd.
www.kichwacoders.com

On 18 March 2016 at 19:53, Alena Laskavaia <elaskavaia.cdt@xxxxxxxxx> wrote:
Thanks Marc.

If you not dealing with gdb tests you can skip the rest

The new structure user Parametrized junit4 "Runner" to inject gdb version in all tests that are gdb based.
The following changes were made:
All Suite_X  and SuiteRemote_X were removed and consolidated in one suite called SuiteGdb. In you adding new test case add it there.
There are also 3 wrapper suites to launch that:
- AutomatedSuite - this what currently run from maven. This is parametrized by java variable "cdt.tests.dsf.gdb.versions", which is now set from pom file and you can set it from IDE (see below how)
- AllSuites - will run all gdbs it can find, ignore all it can't, that will ignore setting of cdt.tests.dsf.gdb.versions
- AllRemoteSuites - same as above but runs all tests with gdbserver

All tests which were inhering one another were merged into one, its uses assume function to run the right tests.

I moved non-stop tests to subpackage just to reduce number of test cases on top level but you are welcome to restructure it as you like

Since injected version is a field variable I cannot use global overrides which was used before. So all version testing has to be done in test (of @Before)
not in @BeforeClass methods. Overhead of skipping a test is 20ms which is reasonable considered that running test takes at minium 0.5 second.
I can do a bit more re-structuring later to optimize more.


Running from IDE
1) Since I replaced the runner to Parameterized and junit can only have one running I cannot use Background runner anymore. Since IDE by default
is trying to run tests in UI thread that won't be good for these. You either have to go to main tab and uncheck "Run in UI thread" or use "Headless" application
TODO: fail test if running in UI thread right away

2) You can run either individual test or any suite provided you did #1. By default parameters are "gdb,gdbserver", so if you don't change anything it will
run you default gdb and gdbserver binaries. It gets the actual version from running them to properly run "assumes". If you want to change this you need to add arguments.
Like -Dcdt.tests.dsf.gdb.versions=gdb.7.7,gdbserver.7.7,gdb.7.11 in "VM Arguments" section of launch config on Arguments tab.
So this is command separated list of gdb versions (without spaces) with "gdb" prefix for gdb local and gdbserver prefix for remote (note if you want to run all just use AllSuites instead)

Developing new test cases
- If you adding test case in existing ones just make sure you use assume for the right gdb
If some functionality only available (or not) in certain gdb's you can use junit4 assume statement.
Parent class provides that following functions for convenience
assumeGdbVersionAtLeast("7.4") - test will only be run for gdb 7.4 or higher and ignored for the rest
assumeGdbVersionLowerThen("6.9") - test will be run only for gdb which is stricly lower than 6.9 (which is 6.8 and lower)
assumeGdbVersionNot("6.8") - skip the test for a specific gdb.
You can also combine them if need be
You can also assume for remote if you test only remote (or visa versa), i.e        
Assume.assumeTrue("Skipping non-remote", remote);

If you need to change some setup function based on version there are some version comparision utils you can use. I only found 2 method then needed if version
for setup


- If you adding new test case make sure it extends BaseParametrizedTestCase and use Parametrized running (see others as example)

- And new test case into SuiteGdb (remote or local)

Implementation details:
if you looking for implementation details interesting classes are BaseParametrizedTestCase which defines getVersion which is what it injected as parameters,
AllSuite - how you can override parameters from a class (cannot do it if using Suite annotations)

TODO:
- I should put this test on the wiki somewhere
- Lots of tests are using sleeps which is bad for performance and random failures
- Lots of tests use similar harness which can be implemented as junit4 Rules
- We can make a smart default run which instead of running specific version of gdb will run all versions but only tests which won't be run in any higher version, i.e. all tests will be run once if most approprate gdb version

P.S. Thanks Marc for putting up my re-formatting (although it was not reformatted but white space removing tool but it did something awful in some cases).



On Fri, Mar 18, 2016 at 11:00 AM, Marc Khouzam <marc.khouzam@xxxxxxxxxxxx> wrote:
I want to thank Elena for her recent efforts to cleanup the debug test suite.
We had a lot of classes simply to have different names for different gdb versions.
She used her JUnit expertise to clean that up and remove 89% of the test classes,
while keeping the same number of actual tests!

Not only will it bec easier to maintain, but it makes it much easier when a new GDB version
comes out.

I'll let her explain the new way to launch the tests, to make sure I also fully understand :)

Nice job Elena!

Marc


From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Alena Laskavaia [elaskavaia.cdt@xxxxxxxxx]
Sent: March 17, 2016 10:42 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] New DSF test structure

I merged the SourceLookup test changes already in my next commit.
Its important to make writing junits easy. Once I am done refactoring I will write some info on how to use it from now on on dev list.

On Wed, Mar 16, 2016 at 7:04 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:
Hi Alena,

The new test structure [1] looks great. I will update the
SourceLookupTest once my big outstanding gerrits that touch them them
are merged.

I think it is great how with this structure I can easily run one test
on all GDB versions as a single JUnit run without having to create a
manual suite as I have been doing up until now [2]. It also looks
great in the JUnit UI.

Thanks!
Jonah


[1] https://mattermost-test.eclipse.org/eclipse/pl/1yphynpzn7g4upk4sbr5kixykh
and https://git.eclipse.org/r/#/c/68371/

[2] https://git.eclipse.org/r/#/c/52068/1/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/SourceLookupSuite.java
- this file is not part of the review anymore, but only public link I
had to it.

~~~
Jonah Graham
Kichwa Coders Ltd.
www.kichwacoders.com
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev


Back to the top