Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [aspectj-dev] Spring cleaning in the tests department.... (committers pls read)

Hi Adrian -

A quick reply before I look at these updates in detail.

This is nice, but the functionality already existed in
HarnessJUnitUtil and AjctestsAdapter to run tests
using JUnit.  (See my dev email, attached)  As I
remember, it wasn't recommended only because JUnit was
a lot slower.  (Other features, like running only specific
tests, were also supported by the harness.)  

As for writing test cases using JUnit/AjcTestCase, we
specify test cases (in .xml or .txt files) rather than 
writing them programmatically so the interpreter can handle
any changes and also reuse the tests.  For example, you can
run all the old tests with new options, or run them using
Ant, AJDE, etc.  (We've been considering an adapter that 
runs source tests as binary-weaving tests.)  By contrast, 
the tests written programmatically tend to break with 
each change of options or behavior.  When the specs must
be updated, it's typically easier to do in the spec file
than in programmatic tests.  So it's a win for reusability
and maintainability.

Now for the original problem (read: itch): there was something
you wanted to do that you couldn't, and upgrading the harness
to make that work seemed like too much effort.  While the
harness should support checks between incremental steps of
the output (black-box), it doesn't support more white-box
checks (e.g., of changes to the in-memory structure model).
One way to address that is to support a plug point for a 
component which is handed the compiler between steps, for
programmatic verification.  Another is to write a step that
verifies the structure model based on specifications.  I'll
look at your incremental test case and see what I can do.

So let's talk about how to get what you want (JUnit interface,
support for writing incremental tests) without duplicating 
effort or going too far down the road of programmatic tests.  
I can join this week's AJDT call, if you would like to talk 
about it then.

Wes

P.S. - re:
> For new tests that you add as part of the 1.2.1 development, please add 
> the
> tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file, and 
> to
> the org.aspectj.systemtest.ajc121.Ajc121Tests test case class. 

A small caveat wrt test suite organization (using 
ajc121-tests.xml):  breaking out .xml files is fine (as I remember 
the harness handles inclusion fine), but we shouldn't use the suite 
file as a kind of global tag for the included tests.  
We mark tests with language versions 
to exclude them from test runs with earlier versions; conversely,
tests written during a given release but valid for other versions
can still be run when testing those versions.  So any test that
requires 1.2.1 should be marked as such, but others shouldn't.

I'm also uncomfortable having two definitions of "suite", the
JUnit suite file and the xml file, since it seems like they would
get out of sync pretty easily.  Does that mean having to update
JUnit files when removing or updating test cases?  hmm.

> ------------Original Message------------
> From: Adrian Colyer <adrian_colyer@xxxxxxxxxx>
> To: aspectj-dev@xxxxxxxxxxxxxxx
> Date: Wed, Aug-4-2004 6:24 AM
> Subject: [aspectj-dev] Spring cleaning in the tests department.... (committers pls read)
>
> 
> 
> 
> 
> It all started out simply enough. I wanted to write some unit tests for 
> a
> bug I'd been working on, and the test harness didn't quite support what 
> I
> wanted to do (or at least I couldn't figure out how to make it do what 
> I
> wanted it to do - one of the two anyway).
> 
> I wrote a helper class org.aspectj.tools.ajc.AjcTestCase which is now 
> in
> org.aspectj.ajdt.core/testsrc/ . This serves as a base class for any
> programmatic JUnit test you want to write to drive the compiler. In the
> same package under the testsrc folder are the helper classes it uses - 
> Ajc
> (to drive normal and incremental compiles - giving you full control 
> over
> when the next increment should occur), the CompilationResult class, and 
> a
> couple of others. I javadoc'd them as i think they're generally useful 
> -
> the doc is in tests/doc (the tests module).
> 
> Well, it was only a short step from there...
> ....to write another TestCase class that extends AjcTestCase, called
> XMLBasedAjcTestCase. Given a suite file (of the ajcTests.xml format), 
> it
> allows you to run any individual test under JUnit.  This enabled me to 
> do
> some tidy up of the tests in the tests module. I divided the tests in
> ajcTests.xml up into logical batches (using some archeology to help) - 
> they
> can now be found in src/org.aspectj.systemtest.xxx/xxx.xml, where xxx 
> is
> one of:
> 
> ajc121,ajc120,ajc11,ajc10x,pre10x,base,design,incremental,serialVerUID,xlint,options,aspectpath,inpath,java14,pureJava,
> knownLimitations
> 
> These files are all included by a now much shorter (textually) 
> ajcTests.xml
> file so that you can run the harness as normal - but there's also now 
> an
> alternative to this mode of operation: In each of the
> org.aspectj.systemtest.xxx packages is a JUnit test case class that 
> runs
> all the tests in that suite. These are bundled together into  AllTests 
> (a
> suite that can run under JDK 1.3), and AllTests14 (includes all the 1.3
> tests plus the 1.4 specific ones). By launching one of these using 
> JUnit
> you can run all of the tests - with each individual ajc-test being it's 
> own
> JUnit test (for existing xml spec files, I wrote a class MakeTestCase 
> in
> the testing module that generates the JUnit test case classes, you 
> should
> never need to run this though because I've done them all).  This mode 
> of
> operation has a number of advantages:
> 
> * integrates better into Eclipse
> * easy to see which tests have failed (normal JUnit view)
> * easy to re-run or debug any failed test
> * easy to run any individual test
> 
> all just using the normal JUnit facilities.
> 
> In run-all-junit-modules I checked in the Suite
> "RunTheseBeforeYouCommitTests" that runs all the unit tests from all 
> the
> modules (including the ones from 'tests') - try it, you'll like it - 
> much
> easier to work with.
> 
> I wrote a much simpler and smaller set of classes to parse the ajc 
> suite
> files (still using the Digester) - these don't have the full checking 
> and
> sophisticated option handling of the harness ones, but they do contain 
> all
> of the function needed to run every single test case in the suite. The
> exception is the incremental tests - I didn't add support for 
> inc-compile.
> Instead take a look at 
> org.aspectj.systemtest.incremental.IncrementalTests
> which demonstrates the really nice feature of being able to mingle
> XML-based and programmatic testing in the same test. This makes the
> incremental tests much easier to write and will finally enable me to 
> write
> some of the incremental tests I've been putting off since 1.2,
> 
> For new tests that you add as part of the 1.2.1 development, please add 
> the
> tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file, and 
> to
> the org.aspectj.systemtest.ajc121.Ajc121Tests test case class. Since 
> you
> can easily run individual tests from JUnit, this should alleviate the 
> need
> to maintain your own xxxTests.xml files and cut-and-paste tests to and 
> from
> ajcTests.xml all the time.
> 
> On the way, I trimmed out a few tests in ajcTests.xml that were 
> pure-java
> tests (and that didn't have the pureJava keyword). These are all in the
> suite PureJavaTests.xml, along with the ones that did have the keyword.
> They are *not* included in the default suites, but with a little 
> tidying up
> (of the tests with the 'pureJava' keyword, that clearly haven't been 
> run
> for a while), these could be added to a release build test suite.
> 
> -- Adrian.
> adrian_colyer@xxxxxxxxxx
> 
> 
> 
> 
> _______________________________________________
> aspectj-dev mailing list
> aspectj-dev@xxxxxxxxxxx
> http://dev.eclipse.org/mailman/listinfo/aspectj-dev
> 
--- Begin Message ---
  • From: Wes Isberg <wes@xxxxxxxxxxxxxx>
  • Date: Fri, 29 Aug 2003 01:07:10 -0700
  • User-agent: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
(A comment below to make clear a check-in from last week.)

Also, I'm interested in the cruise-control machine running
the full release build/test cycle, which includes JDK variants,
product tests, etc.  They take hours on my machine (while
a run through all the JUnit and harness tests takes minutes).
Any chance of that happening, with a pointer to most-recent
results?

Adrian Colyer wrote:

To support automatic execution of the full suite of AspectJ tests as part of the cruise-control builds, I've made the following changes/additions to the test harness:

* I extended testing.drivers.Harness with an additional output formatter so that it can produce junit XML formatted test results.

I like this idea.  Another way is to wrap the tests as JUnit tests and
use JUnit to run them.  Last week I updated the code we had for that.
To create a suite programmatically, use HarnessJUnitUtil.suite(...).[1]
To run ajcTests.xml, use AjcTestsUsingJUnit.  You might prefer these
to the output formatter because you can run them in any JUnit GUI
(e.g., in Eclipse) and because they stuff all the test results into
the assertion message, so you can analyze failures without re-running
the tests.  (However, neither this solution nor the xml outputter
properly segregates stderr/stdout as the harness can.)

Thanks -
Wes

[1] This and other files are in testsrc/org/aspectj/testing/drivers.


<snip>


--- End Message ---

Back to the top