A
discussion about verifying TCKs was cut short on the Google
docs, and that's probably because it's hard to follow
discussions in the tiny comments boxes, so I wanted to start a
thread here.
Firstly,
I'm not suggesting that Eclipse run the TCKs themselves, that's
obviously not feasible.
The
problem is that there is an assumption that just because the
TCKs are open, anyone can run them, and so therefore anyone can
verify whether a vendor is falsifying test reports. This is not
true. For example, most MicroProfile TCKs are built on
Arquillian, which is a testing tool that abstracts away an
application server, allowing specs to produce vendor agnostic
TCKs. Arquillian itself is open, however, the integration with a
particular app server is not part of Arquillian, and may or may
not be open. If the Arquillian integration for a particular app
server is not open and not freely available, then only people
that have access to the Arquillian integration for that app
server will be able to run the TCK against it. This means, in
such a case, it won't be possible for anyone to verify whether
the TCK passed against that app server. Even if the app server
vendor provides the binaries for Arquillian integration - we
need more than that, we need the source code of the Arquillian
integration to ensure that they haven't implemented any smoke
and mirrors in their integration. For example, what's to stop
them from providing Arquillian integration that actually
integrates to a different vendors app server instead of theirs,
in that case, the TCK might pass, but only because it was
testing a different app server.
So
I think any implementation that wishes to claim compatibility
must, in addition to publishing test results, also publish any
integration code necessary to run the TCK as open source. I
would go as far as to say that they should also provide scripts
and detailed instructions saying how to run the TCK, since for
some vendors app servers this might not be at all trivial.
Thanks for this. I now understand what you're trying to say, and
agree that it is a valid point.
I wonder if anyone from Oracle could chime in here and comment on
how this is handled currently with Java EE certification?