Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-dev] Jetty Distribution Testing and Remote Test Suites

A few of us have been focusing on validation testing surrounding the
jetty distributions lately and I wanted to update folks on what we
have now, how it working out and solicit some feedback on it as well.

Joakim and I were kicking around the concept of this validation
testing and hit upon the idea of what could be termed a remote assert
when your looking at things from a unit testing perspective.  Joakim
being Joakim scurried into a corner and massaged junit to work inside
of a servlet context and jetty-remote-assert was born.

The 10k foot view of this system envisioned is:

We can have a series of war files that all have a common servlet mount
point, say something like this

http://localhost:8080/test-war-download/testSuite

If you navigated to this url it will return a mess of JSON that
describes the results of running tests from the servlet.  These these
are actually just like normal junit tests, right down to being pojo's
that leverage junit annotations.  @Ignore, @Test, @Before, @After,
etc...

The servlet mounted at testSuite is dead simple, extending the
RemoteAssertServlet and looks like this:

public class TestSuiteServlet extends RemoteAssertServlet
{
  public TestSuiteServlet()
  {
    addTestClass(DownloadTest.class);
  }
}

And the DownloadTest class looks like this:

public class DownloadTest
{
    @Rule
    public ServletRequestContextRule context = new ServletRequestContextRule();

    @Test
    public void testDownloadSmall()
    {
        Assert.assertTrue("example", true);
    }
}

Inside of the test you can use a simple http connection, or the jetty
client, or whatever you like to do pretty much whatever you want from
the remote unit test.  Hitting the testSuite context will run all test
classes that are registered with this test suite servlet and the
resulting test results will be sent back on the response as a json
string.  The @Rule will give your pojo access to the Request,
Response, and Servlet if you so need it.

[
 {
   "name": "org.mortbay.jetty.tests.download.tests.DownloadTest",
   "results": [{
     "className": "org.mortbay.jetty.tests.download.tests.DownloadTest",
     "methodName": "testDownloadSmall",
     "success": true
   }],
   "testCount": 1
 },
...
]

If there are more then one test you can see in the format how that
works, and multiple test classes will just manifest more blocks of
json output.  Exceptions and failed tests will come back in the json
as well to facilitate debugging and pinpointing which remote assert
failed.

On the client side of this whole testing ordeal, we can iterate
through that JSON and validate that all remote tests were
successful....we have a working example in the test-war-download
artifact where there is now a pojo that has @Test annotations that
each attempt to download a file from itself really...normal client
connection.  So its a bit like a self contained self-referencial
webapp, where you hit this testSuite context and get back a block of
json.

There is also support in the RemoteAssertServlet to invoke a specific
test class or a specific test within a test class, also returning a
block of json.

So, all that is nice and all but what does it all mean.  Currently we
have a large number of unit tests, ever increasing in jetty proper for
all manner of situations as we should since we don't specifically run
a distribution, embedded is a huge user.  But when we push out a
distribution right now we are sort of at the mercy of our click
testing, downloading the distribution and manually validating that
everything works...and with the variety of configurations we like to
test we rarely can test it all from the distribution perspective.  So
the longer term goal with this mechanism is to have all this
functionality for all the
things we generally test in jetty distribution by hand...jsp support,
cometd support, etc etc...all automatically tested when we are
building a release so we have a higher degree of confidence in the
release.

There is a test-jetty-distribution module on codehaus now that has
working examples of how all this works.  Using a JettyDistro class in
the jetty-test-helper we can easily build test cases that do the
manual configuring of the distribution (enabling jndi bits, tweaking
the start.ini, etc) all in a unit test, and then start up the
distribution, run our tests, shut it down, and move on to the next
test case.  I have this working for the test-war-download that is
mentioned above.  Michael wrote the webapp and I updated it to use the
jetty-remote-assert, so it now tests for us the simple case in the
test-jetty-webapp on the eclipse side.  Downloading 4 different sizes
of files and makes sure that they are cleanly obtained.  Simple case
sure but was previously 4 clicks in manual validation testing, and
even then we didn't technically validate the files were exactly the
same.

I suspect we can leverage some of our existing unit tests for things
like cometd and other bits of jetty to massage test-war-* artifacts
that can do much more in-depth testing and validation.  We have
knocked around a test-war-sessions artifact that would let you test
out the common session usage scenarios to make sure they are working
correctly within the distribution.

Note: most all of this is being done on the codehaus side since we
don't have to mess with CQ's and whatnot in this early stage

Anyway, that is it in a nutshell, any thoughts or feedback on the
value of this?  Do we keep it and expand on it or are we fine with
most of the in-artifact testing that we currently do for functionality
and the whole idea of testing the distribution as a separate entity is
not important enough to warrant this?

cheers,
jesse

--
jesse mcconnell
jesse.mcconnell@xxxxxxxxx


Back to the top