[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [aspectj-dev] Spring cleaning in the tests department.... (committers pls read)

> This is nice, but the functionality already existed in
> HarnessJUnitUtil and AjctestsAdapter to run tests
> using JUnit.  (See my dev email, attached)  As I
> remember, it wasn't recommended only because JUnit was
> a lot slower.  (Other features, like running only specific
> tests, were also supported by the harness.) 

Doh! And such a fun day I had too.... ;).
I just tried out the AjcTestsUsingJUnit and it works very nicely. A 
combination of what you've already done and what I just implemented would 
be perfect. The key thing that AjcTestsUsingJUnit doesn't give me is the 
ability to rerun any failed test (unless I run the whole suite in debug 
mode and elect to 'Keep JUnit running' - but that *is* slow). Being able 
to instantly run (or re-run) any individual test (failing or otherwise) is 
one of the key things I was after. The solution I implemented gives you 
that but has one key disadvantage :- the test classes are generated from 
the spec files. The actual test specifications remain in the XML (no 
disagreement from me here on the value of that) - the only thing generated 
is a one line test method that says "runTest("test title");". I didn't 
give the actual tests good names either (so you see test001, test002, ... 
go by in JUnit - could fix the generator for that if it turned out to be a 
real issue though).
Generating the test case classes like this shouldn't be an issue for the 
existing tests that we run (and run, and run, and run), but don't change. 
For new tests it is more of a pain because you *do* have to add the 
one-line method to the test suite class in order to drive it. Now you see 
another reason for my splitting ajcTests.xml into a set of constituent 
files - the "ajc121.xml" spec file is not intended to mean these tests 
only run under 1.2.1, it is meant to simply mean "these are the tests we 
added for bugs and enhancements addressed in the 1.2.1 release".  The 
existing keyword mechanism in the suite.xml files should continue to be 
used to indicate which tests work in which environments :- ajcTests.xml is 
actually no different now (once the XML processor has handled the 
includes) than it was before.

I agree that we should use the textual specification in the *.xml files 
whenever possible to define a test, but sometimes it *is* useful to be 
able to add some programmatic logic (otherwise where do you stop - you 
could end up with a whole XML programming language!). The solution I put 
together lets you do this by calling "runTest('test title") to do the 
test-by-specification piece, but also allowing code before or after that 
(if you really need it) to do additional testing. I found that facility to 
be very useful when writing incremental tests, and Andy also used the 
existence of the test case class to put some logic into the setup() and 
teardown() methods that he needed for his incremental model tests. 

We can use the first part of the call today to work out the best overall 
approach as you suggest,
Cheers, Adrian.

-- Adrian

"Wes Isberg" <wes@xxxxxxxxxxxxxx> 
Sent by: aspectj-dev-admin@xxxxxxxxxxx
04/08/2004 20:15
Please respond to


Re: [aspectj-dev] Spring cleaning in the tests department.... (committers 
pls read)

Hi Adrian -

A quick reply before I look at these updates in detail.

This is nice, but the functionality already existed in
HarnessJUnitUtil and AjctestsAdapter to run tests
using JUnit.  (See my dev email, attached)  As I
remember, it wasn't recommended only because JUnit was
a lot slower.  (Other features, like running only specific
tests, were also supported by the harness.) 

As for writing test cases using JUnit/AjcTestCase, we
specify test cases (in .xml or .txt files) rather than 
writing them programmatically so the interpreter can handle
any changes and also reuse the tests.  For example, you can
run all the old tests with new options, or run them using
Ant, AJDE, etc.  (We've been considering an adapter that 
runs source tests as binary-weaving tests.)  By contrast, 
the tests written programmatically tend to break with 
each change of options or behavior.  When the specs must
be updated, it's typically easier to do in the spec file
than in programmatic tests.  So it's a win for reusability
and maintainability.

Now for the original problem (read: itch): there was something
you wanted to do that you couldn't, and upgrading the harness
to make that work seemed like too much effort.  While the
harness should support checks between incremental steps of
the output (black-box), it doesn't support more white-box
checks (e.g., of changes to the in-memory structure model).
One way to address that is to support a plug point for a 
component which is handed the compiler between steps, for
programmatic verification.  Another is to write a step that
verifies the structure model based on specifications.  I'll
look at your incremental test case and see what I can do.

So let's talk about how to get what you want (JUnit interface,
support for writing incremental tests) without duplicating 
effort or going too far down the road of programmatic tests. 
I can join this week's AJDT call, if you would like to talk 
about it then.


P.S. - re:
> For new tests that you add as part of the 1.2.1 development, please add 
> the
> tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file, and 
> to
> the org.aspectj.systemtest.ajc121.Ajc121Tests test case class. 

A small caveat wrt test suite organization (using 
ajc121-tests.xml):  breaking out .xml files is fine (as I remember 
the harness handles inclusion fine), but we shouldn't use the suite 
file as a kind of global tag for the included tests. 
We mark tests with language versions 
to exclude them from test runs with earlier versions; conversely,
tests written during a given release but valid for other versions
can still be run when testing those versions.  So any test that
requires 1.2.1 should be marked as such, but others shouldn't.

I'm also uncomfortable having two definitions of "suite", the
JUnit suite file and the xml file, since it seems like they would
get out of sync pretty easily.  Does that mean having to update
JUnit files when removing or updating test cases?  hmm.

> ------------Original Message------------
> From: Adrian Colyer <adrian_colyer@xxxxxxxxxx>
> To: aspectj-dev@xxxxxxxxxxxxxxx
> Date: Wed, Aug-4-2004 6:24 AM
> Subject: [aspectj-dev] Spring cleaning in the tests department.... 
(committers pls read)
> It all started out simply enough. I wanted to write some unit tests for 
> a
> bug I'd been working on, and the test harness didn't quite support what 
> I
> wanted to do (or at least I couldn't figure out how to make it do what 
> I
> wanted it to do - one of the two anyway).
> I wrote a helper class org.aspectj.tools.ajc.AjcTestCase which is now 
> in
> org.aspectj.ajdt.core/testsrc/ . This serves as a base class for any
> programmatic JUnit test you want to write to drive the compiler. In the
> same package under the testsrc folder are the helper classes it uses - 
> Ajc
> (to drive normal and incremental compiles - giving you full control 
> over
> when the next increment should occur), the CompilationResult class, and 
> a
> couple of others. I javadoc'd them as i think they're generally useful 
> -
> the doc is in tests/doc (the tests module).
> Well, it was only a short step from there...
> ....to write another TestCase class that extends AjcTestCase, called
> XMLBasedAjcTestCase. Given a suite file (of the ajcTests.xml format), 
> it
> allows you to run any individual test under JUnit.  This enabled me to 
> do
> some tidy up of the tests in the tests module. I divided the tests in
> ajcTests.xml up into logical batches (using some archeology to help) - 
> they
> can now be found in src/org.aspectj.systemtest.xxx/xxx.xml, where xxx 
> is
> one of:
> knownLimitations
> These files are all included by a now much shorter (textually) 
> ajcTests.xml
> file so that you can run the harness as normal - but there's also now 
> an
> alternative to this mode of operation: In each of the
> org.aspectj.systemtest.xxx packages is a JUnit test case class that 
> runs
> all the tests in that suite. These are bundled together into  AllTests 
> (a
> suite that can run under JDK 1.3), and AllTests14 (includes all the 1.3
> tests plus the 1.4 specific ones). By launching one of these using 
> JUnit
> you can run all of the tests - with each individual ajc-test being it's 
> own
> JUnit test (for existing xml spec files, I wrote a class MakeTestCase 
> in
> the testing module that generates the JUnit test case classes, you 
> should
> never need to run this though because I've done them all).  This mode 
> of
> operation has a number of advantages:
> * integrates better into Eclipse
> * easy to see which tests have failed (normal JUnit view)
> * easy to re-run or debug any failed test
> * easy to run any individual test
> all just using the normal JUnit facilities.
> In run-all-junit-modules I checked in the Suite
> "RunTheseBeforeYouCommitTests" that runs all the unit tests from all 
> the
> modules (including the ones from 'tests') - try it, you'll like it - 
> much
> easier to work with.
> I wrote a much simpler and smaller set of classes to parse the ajc 
> suite
> files (still using the Digester) - these don't have the full checking 
> and
> sophisticated option handling of the harness ones, but they do contain 
> all
> of the function needed to run every single test case in the suite. The
> exception is the incremental tests - I didn't add support for 
> inc-compile.
> Instead take a look at 
> org.aspectj.systemtest.incremental.IncrementalTests
> which demonstrates the really nice feature of being able to mingle
> XML-based and programmatic testing in the same test. This makes the
> incremental tests much easier to write and will finally enable me to 
> write
> some of the incremental tests I've been putting off since 1.2,
> For new tests that you add as part of the 1.2.1 development, please add 
> the
> tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file, and 
> to
> the org.aspectj.systemtest.ajc121.Ajc121Tests test case class. Since 
> you
> can easily run individual tests from JUnit, this should alleviate the 
> need
> to maintain your own xxxTests.xml files and cut-and-paste tests to and 
> from
> ajcTests.xml all the time.
> On the way, I trimmed out a few tests in ajcTests.xml that were 
> pure-java
> tests (and that didn't have the pureJava keyword). These are all in the
> suite PureJavaTests.xml, along with the ones that did have the keyword.
> They are *not* included in the default suites, but with a little 
> tidying up
> (of the tests with the 'pureJava' keyword, that clearly haven't been 
> run
> for a while), these could be added to a release build test suite.
> -- Adrian.
> adrian_colyer@xxxxxxxxxx
> _______________________________________________
> aspectj-dev mailing list
> aspectj-dev@xxxxxxxxxxx
> http://dev.eclipse.org/mailman/listinfo/aspectj-dev

----- Message from Wes Isberg <wes@xxxxxxxxxxxxxx> on Fri, 29 Aug 2003 
01:07:10 -0700 -----
Re: [aspectj-dev] Changes to AspectJ test suite...

(A comment below to make clear a check-in from last week.)

Also, I'm interested in the cruise-control machine running
the full release build/test cycle, which includes JDK variants,
product tests, etc.  They take hours on my machine (while
a run through all the JUnit and harness tests takes minutes).
Any chance of that happening, with a pointer to most-recent

Adrian Colyer wrote:

> To support automatic execution of the full suite of AspectJ tests as 
> of the cruise-control builds, I've made the following changes/additions 
> the test harness:
> * I extended testing.drivers.Harness with an additional output formatter 

> so that it can produce junit XML formatted test results. 

I like this idea.  Another way is to wrap the tests as JUnit tests and
use JUnit to run them.  Last week I updated the code we had for that.
To create a suite programmatically, use HarnessJUnitUtil.suite(...).[1]
To run ajcTests.xml, use AjcTestsUsingJUnit.  You might prefer these
to the output formatter because you can run them in any JUnit GUI
(e.g., in Eclipse) and because they stuff all the test results into
the assertion message, so you can analyze failures without re-running
the tests.  (However, neither this solution nor the xml outputter
properly segregates stderr/stdout as the harness can.)

Thanks -

[1] This and other files are in testsrc/org/aspectj/testing/drivers.