[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [aspectj-dev] Spring cleaning in the tests department.... (committers pls read)
|
Hi Adrian -
Sorry I wasn't on the AJDT call - I needed more notice *smile*
I'll try to integrate what you've done.
(btw, right now, to re-run failed tests I have the harness output the
XML specs for the failures and then just run that output.
To run (one) selected tests, I just list the title(s) in a file and
use the select-by-title keyword (which I've forgotten at the moment.
And I've long intended to send the failing pure-java cases to the
JDT team...)
Re: having to manually update JUnit files for existing and new cases,
I once wrote code for generating test cases in the suite() method by
producing bytecode classes which any JUnit harness could then load.
That avoided having to sync sources, so perhaps I'll do that here.
More importantly, I agree we shouldn't make a whole new (XML-based)
language for incremental or other new kinds of tests, and that it
should be easy to define new tests, so I'll look at supporting
what you've done. I like the support for 'runTest("title")'
because it actually reuses the tests in Java. Good stuff.
Wes
> ------------Original Message------------
> From: Adrian Colyer <adrian_colyer@xxxxxxxxxx>
> To: aspectj-dev@xxxxxxxxxxx
> Date: Thu, Aug-5-2004 7:39 AM
> Subject: Re: [aspectj-dev] Spring cleaning in the tests department.... (committers pls read)
>
> > This is nice, but the functionality already existed in
> > HarnessJUnitUtil and AjctestsAdapter to run tests
> > using JUnit. (See my dev email, attached) As I
> > remember, it wasn't recommended only because JUnit was
> > a lot slower. (Other features, like running only specific
> > tests, were also supported by the harness.)
>
> Doh! And such a fun day I had too.... ;).
> I just tried out the AjcTestsUsingJUnit and it works very nicely. A
> combination of what you've already done and what I just implemented
> would
> be perfect. The key thing that AjcTestsUsingJUnit doesn't give me is
> the
> ability to rerun any failed test (unless I run the whole suite in debug
>
> mode and elect to 'Keep JUnit running' - but that *is* slow). Being
> able
> to instantly run (or re-run) any individual test (failing or otherwise)
> is
> one of the key things I was after. The solution I implemented gives you
>
> that but has one key disadvantage :- the test classes are generated
> from
> the spec files. The actual test specifications remain in the XML (no
> disagreement from me here on the value of that) - the only thing
> generated
> is a one line test method that says "runTest("test title");". I didn't
> give the actual tests good names either (so you see test001, test002,
> ...
> go by in JUnit - could fix the generator for that if it turned out to
> be a
> real issue though).
>
> Generating the test case classes like this shouldn't be an issue for
> the
> existing tests that we run (and run, and run, and run), but don't
> change.
> For new tests it is more of a pain because you *do* have to add the
> one-line method to the test suite class in order to drive it. Now you
> see
> another reason for my splitting ajcTests.xml into a set of constituent
> files - the "ajc121.xml" spec file is not intended to mean these tests
> only run under 1.2.1, it is meant to simply mean "these are the tests
> we
> added for bugs and enhancements addressed in the 1.2.1 release". The
> existing keyword mechanism in the suite.xml files should continue to be
>
> used to indicate which tests work in which environments :- ajcTests.xml
> is
> actually no different now (once the XML processor has handled the
> includes) than it was before.
>
> I agree that we should use the textual specification in the *.xml files
>
> whenever possible to define a test, but sometimes it *is* useful to be
> able to add some programmatic logic (otherwise where do you stop - you
> could end up with a whole XML programming language!). The solution I
> put
> together lets you do this by calling "runTest('test title") to do the
> test-by-specification piece, but also allowing code before or after
> that
> (if you really need it) to do additional testing. I found that facility
> to
> be very useful when writing incremental tests, and Andy also used the
> existence of the test case class to put some logic into the setup() and
>
> teardown() methods that he needed for his incremental model tests.
>
> We can use the first part of the call today to work out the best
> overall
> approach as you suggest,
> Cheers, Adrian.
>
> -- Adrian
> Adrian_Colyer@xxxxxxxxxx
>
>
>
> "Wes Isberg" <wes@xxxxxxxxxxxxxx>
> Sent by: aspectj-dev-admin@xxxxxxxxxxx
> 04/08/2004 20:15
> Please respond to
> aspectj-dev
>
>
> To
> aspectj-dev@xxxxxxxxxxx
> cc
>
> Subject
> Re: [aspectj-dev] Spring cleaning in the tests department....
> (committers
> pls read)
>
>
>
>
>
>
> Hi Adrian -
>
> A quick reply before I look at these updates in detail.
>
> This is nice, but the functionality already existed in
> HarnessJUnitUtil and AjctestsAdapter to run tests
> using JUnit. (See my dev email, attached) As I
> remember, it wasn't recommended only because JUnit was
> a lot slower. (Other features, like running only specific
> tests, were also supported by the harness.)
>
> As for writing test cases using JUnit/AjcTestCase, we
> specify test cases (in .xml or .txt files) rather than
> writing them programmatically so the interpreter can handle
> any changes and also reuse the tests. For example, you can
> run all the old tests with new options, or run them using
> Ant, AJDE, etc. (We've been considering an adapter that
> runs source tests as binary-weaving tests.) By contrast,
> the tests written programmatically tend to break with
> each change of options or behavior. When the specs must
> be updated, it's typically easier to do in the spec file
> than in programmatic tests. So it's a win for reusability
> and maintainability.
>
> Now for the original problem (read: itch): there was something
> you wanted to do that you couldn't, and upgrading the harness
> to make that work seemed like too much effort. While the
> harness should support checks between incremental steps of
> the output (black-box), it doesn't support more white-box
> checks (e.g., of changes to the in-memory structure model).
> One way to address that is to support a plug point for a
> component which is handed the compiler between steps, for
> programmatic verification. Another is to write a step that
> verifies the structure model based on specifications. I'll
> look at your incremental test case and see what I can do.
>
> So let's talk about how to get what you want (JUnit interface,
> support for writing incremental tests) without duplicating
> effort or going too far down the road of programmatic tests.
> I can join this week's AJDT call, if you would like to talk
> about it then.
>
> Wes
>
> P.S. - re:
> > For new tests that you add as part of the 1.2.1 development, please
> add
> > the
> > tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file,
> and
> > to
> > the org.aspectj.systemtest.ajc121.Ajc121Tests test case class.
>
> A small caveat wrt test suite organization (using
> ajc121-tests.xml): breaking out .xml files is fine (as I remember
> the harness handles inclusion fine), but we shouldn't use the suite
> file as a kind of global tag for the included tests.
> We mark tests with language versions
> to exclude them from test runs with earlier versions; conversely,
> tests written during a given release but valid for other versions
> can still be run when testing those versions. So any test that
> requires 1.2.1 should be marked as such, but others shouldn't.
>
> I'm also uncomfortable having two definitions of "suite", the
> JUnit suite file and the xml file, since it seems like they would
> get out of sync pretty easily. Does that mean having to update
> JUnit files when removing or updating test cases? hmm.
>
> > ------------Original Message------------
> > From: Adrian Colyer <adrian_colyer@xxxxxxxxxx>
> > To: aspectj-dev@xxxxxxxxxxxxxxx
> > Date: Wed, Aug-4-2004 6:24 AM
> > Subject: [aspectj-dev] Spring cleaning in the tests department....
> (committers pls read)
> >
> >
> >
> >
> >
> > It all started out simply enough. I wanted to write some unit tests
> for
> > a
> > bug I'd been working on, and the test harness didn't quite support
> what
> > I
> > wanted to do (or at least I couldn't figure out how to make it do
> what
> > I
> > wanted it to do - one of the two anyway).
> >
> > I wrote a helper class org.aspectj.tools.ajc.AjcTestCase which is now
>
> > in
> > org.aspectj.ajdt.core/testsrc/ . This serves as a base class for any
> > programmatic JUnit test you want to write to drive the compiler. In
> the
> > same package under the testsrc folder are the helper classes it uses
> -
> > Ajc
> > (to drive normal and incremental compiles - giving you full control
> > over
> > when the next increment should occur), the CompilationResult class,
> and
> > a
> > couple of others. I javadoc'd them as i think they're generally
> useful
> > -
> > the doc is in tests/doc (the tests module).
> >
> > Well, it was only a short step from there...
> > ....to write another TestCase class that extends AjcTestCase, called
> > XMLBasedAjcTestCase. Given a suite file (of the ajcTests.xml format),
>
> > it
> > allows you to run any individual test under JUnit. This enabled me
> to
> > do
> > some tidy up of the tests in the tests module. I divided the tests in
> > ajcTests.xml up into logical batches (using some archeology to help)
> -
> > they
> > can now be found in src/org.aspectj.systemtest.xxx/xxx.xml, where xxx
>
> > is
> > one of:
> >
> >
> ajc121,ajc120,ajc11,ajc10x,pre10x,base,design,incremental,serialVerUID,xlint,options,aspectpath,inpath,java14,pureJava,
> > knownLimitations
> >
> > These files are all included by a now much shorter (textually)
> > ajcTests.xml
> > file so that you can run the harness as normal - but there's also now
>
> > an
> > alternative to this mode of operation: In each of the
> > org.aspectj.systemtest.xxx packages is a JUnit test case class that
> > runs
> > all the tests in that suite. These are bundled together into
> AllTests
> > (a
> > suite that can run under JDK 1.3), and AllTests14 (includes all the
> 1.3
> > tests plus the 1.4 specific ones). By launching one of these using
> > JUnit
> > you can run all of the tests - with each individual ajc-test being
> it's
> > own
> > JUnit test (for existing xml spec files, I wrote a class MakeTestCase
>
> > in
> > the testing module that generates the JUnit test case classes, you
> > should
> > never need to run this though because I've done them all). This mode
>
> > of
> > operation has a number of advantages:
> >
> > * integrates better into Eclipse
> > * easy to see which tests have failed (normal JUnit view)
> > * easy to re-run or debug any failed test
> > * easy to run any individual test
> >
> > all just using the normal JUnit facilities.
> >
> > In run-all-junit-modules I checked in the Suite
> > "RunTheseBeforeYouCommitTests" that runs all the unit tests from all
> > the
> > modules (including the ones from 'tests') - try it, you'll like it -
> > much
> > easier to work with.
> >
> > I wrote a much simpler and smaller set of classes to parse the ajc
> > suite
> > files (still using the Digester) - these don't have the full checking
>
> > and
> > sophisticated option handling of the harness ones, but they do
> contain
> > all
> > of the function needed to run every single test case in the suite.
> The
> > exception is the incremental tests - I didn't add support for
> > inc-compile.
> > Instead take a look at
> > org.aspectj.systemtest.incremental.IncrementalTests
> > which demonstrates the really nice feature of being able to mingle
> > XML-based and programmatic testing in the same test. This makes the
> > incremental tests much easier to write and will finally enable me to
> > write
> > some of the incremental tests I've been putting off since 1.2,
> >
> > For new tests that you add as part of the 1.2.1 development, please
> add
> > the
> > tests into the org.aspectj.systemtest.ajc121/ajc121-tests.xml file,
> and
> > to
> > the org.aspectj.systemtest.ajc121.Ajc121Tests test case class. Since
> > you
> > can easily run individual tests from JUnit, this should alleviate the
>
> > need
> > to maintain your own xxxTests.xml files and cut-and-paste tests to
> and
> > from
> > ajcTests.xml all the time.
> >
> > On the way, I trimmed out a few tests in ajcTests.xml that were
> > pure-java
> > tests (and that didn't have the pureJava keyword). These are all in
> the
> > suite PureJavaTests.xml, along with the ones that did have the
> keyword.
> > They are *not* included in the default suites, but with a little
> > tidying up
> > (of the tests with the 'pureJava' keyword, that clearly haven't been
> > run
> > for a while), these could be added to a release build test suite.
> >
> > -- Adrian.
> > adrian_colyer@xxxxxxxxxx
> >
> >
> >
> >
> > _______________________________________________
> > aspectj-dev mailing list
> > aspectj-dev@xxxxxxxxxxx
> > http://dev.eclipse.org/mailman/listinfo/aspectj-dev
> >
>
>
> ----- Message from Wes Isberg <wes@xxxxxxxxxxxxxx> on Fri, 29 Aug 2003
> 01:07:10 -0700 -----
> To:
> aspectj-dev@xxxxxxxxxxx
> Subject:
> Re: [aspectj-dev] Changes to AspectJ test suite...
>
> (A comment below to make clear a check-in from last week.)
>
> Also, I'm interested in the cruise-control machine running
> the full release build/test cycle, which includes JDK variants,
> product tests, etc. They take hours on my machine (while
> a run through all the JUnit and harness tests takes minutes).
> Any chance of that happening, with a pointer to most-recent
> results?
>
> Adrian Colyer wrote:
>
> > To support automatic execution of the full suite of AspectJ tests as
> part
> > of the cruise-control builds, I've made the following
> changes/additions
> to
> > the test harness:
> >
> > * I extended testing.drivers.Harness with an additional output
> formatter
>
> > so that it can produce junit XML formatted test results.
>
> I like this idea. Another way is to wrap the tests as JUnit tests and
> use JUnit to run them. Last week I updated the code we had for that.
> To create a suite programmatically, use HarnessJUnitUtil.suite(...).[1]
> To run ajcTests.xml, use AjcTestsUsingJUnit. You might prefer these
> to the output formatter because you can run them in any JUnit GUI
> (e.g., in Eclipse) and because they stuff all the test results into
> the assertion message, so you can analyze failures without re-running
> the tests. (However, neither this solution nor the xml outputter
> properly segregates stderr/stdout as the harness can.)
>
> Thanks -
> Wes
>
> [1] This and other files are in testsrc/org/aspectj/testing/drivers.
>
>
> <snip>
>
>
>
> _______________________________________________
> aspectj-dev mailing list
> aspectj-dev@xxxxxxxxxxx
> http://dev.eclipse.org/mailman/listinfo/aspectj-dev
>
