A
discussion about verifying TCKs was cut short on the Google
docs, and that's probably because it's hard to follow
discussions in the tiny comments boxes, so I wanted to start a
thread here.
Firstly,
I'm not suggesting that Eclipse run the TCKs themselves,
that's obviously not feasible.
The
problem is that there is an assumption that just because the
TCKs are open, anyone can run them, and so therefore anyone
can verify whether a vendor is falsifying test reports. This
is not true. For example, most MicroProfile TCKs are built on
Arquillian, which is a testing tool that abstracts away an
application server, allowing specs to produce vendor agnostic
TCKs. Arquillian itself is open, however, the integration with
a particular app server is not part of Arquillian, and may or
may not be open. If the Arquillian integration for a
particular app server is not open and not freely available,
then only people that have access to the Arquillian
integration for that app server will be able to run the TCK
against it. This means, in such a case, it won't be possible
for anyone to verify whether the TCK passed against that app
server. Even if the app server vendor provides the binaries
for Arquillian integration - we need more than that, we need
the source code of the Arquillian integration to ensure that
they haven't implemented any smoke and mirrors in their
integration. For example, what's to stop them from providing
Arquillian integration that actually integrates to a different
vendors app server instead of theirs, in that case, the TCK
might pass, but only because it was testing a different app
server.
So
I think any implementation that wishes to claim compatibility
must, in addition to publishing test results, also publish any
integration code necessary to run the TCK as open source. I
would go as far as to say that they should also provide
scripts and detailed instructions saying how to run the TCK,
since for some vendors app servers this might not be at all
trivial.
Thanks for this. I now understand what you're trying to say,
and agree that it is a valid point.
I wonder if anyone from Oracle could chime in here and comment
on how this is handled currently with Java EE certification?