Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-wg] Verifying TCK compatibility claims

Obviously we're not interested in either extreme.  We're looking for the right cost/benefit tradeoff.

If we think that users are going to be running the TCK against vendor's products because they don't believe the product actually passed the TCK, we have a much bigger problem.

The TCK is a very complex test suite (which doesn't use Arquillian, by the way).  Setting up and running the TCK against someone else's product is a difficult task, even given all the adapters that are needed.  It's far more likely that people will do it wrong and make incorrect claims that the product doesn't pass the TCK.

The TCK adapter for a proprietary product may make use of proprietary APIs that the vendor does not want exposed.

Or the proprietary product may come with a license that prohibits independent testing of the product.

Or the product may only be usable on hardware or in environments that most users won't have access to.

I can imagine many reasons a vendor might not want to make their TCK adapter available to the public or available under an open source license.

All that said, I'm sure many vendors will make their TCK adapters available, and will make their TCK results available, so that others can see exactly how they've tested their products.  But I think there's good reasons that some vendors may not want to do that, and I don't see the benefit of requiring them to do it to be great enough.  The bar is already high enough to become a Java EE / Jakarta EE vendor, we don't need to make it unnecessarily higher.

Of course, if experience makes us suspect that vendors are cheating, we can change the rules and require this.  But in 20 years of Java EE that hasn't been a problem.


James Roper wrote on 07/18/2018 08:39 PM:
On Thu, 19 Jul 2018 at 12:37, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:

If you push this to the extreme, you'll need to see the source code for the vendor's product so you can make sure they haven't implemented some special "TCK mode".  Obviously that's not feasible.

Well, if we're talking about extremes, why not go the other way, and have no TCKs, and just trust each vendor at their word that they've been careful and done their own testing to ensure they implement the spec? But we're not talking about extremes here, so there's no point in bringing it up. Publishing an Arquillian adapter as open source is a low cost, high value thing that can be done, extremes are irrelevant when we're talking about such a cheap win. It also provides a very simple starting point for the frank discussions and compromises that you're saying should be done when necessary, since it means everyone can start from the same meaningful place, "we've run the TCK against product X and found Y". Plus most, if not all Arquillian adapters for Java EE containers out there are already open source, so in most cases, adding such a requirement primarily reinforces what is already done, it doesn't add anything new.

In the end, cheating is dealt with in the courts.  But cheating is rare, and it rarely gets to that point.  Before then, there are plenty of opportunities for frank discussion and compromises when necessary.  Most vendors are trying to do the right thing, and when the issues are explained to them they'll do their best to follow the rules.  That's part of the reason that we have rules that go with the TCK, not just tests, and when you sign the license for the TCK you agree to follow the rules.


Mike Milinkovich wrote on 07/18/2018 06:16 PM:
On 2018-07-18 9:05 PM, James Roper wrote:
A discussion about verifying TCKs was cut short on the Google docs, and that's probably because it's hard to follow discussions in the tiny comments boxes, so I wanted to start a thread here.

Firstly, I'm not suggesting that Eclipse run the TCKs themselves, that's obviously not feasible.

The problem is that there is an assumption that just because the TCKs are open, anyone can run them, and so therefore anyone can verify whether a vendor is falsifying test reports. This is not true. For example, most MicroProfile TCKs are built on Arquillian, which is a testing tool that abstracts away an application server, allowing specs to produce vendor agnostic TCKs. Arquillian itself is open, however, the integration with a particular app server is not part of Arquillian, and may or may not be open. If the Arquillian integration for a particular app server is not open and not freely available, then only people that have access to the Arquillian integration for that app server will be able to run the TCK against it. This means, in such a case, it won't be possible for anyone to verify whether the TCK passed against that app server. Even if the app server vendor provides the binaries for Arquillian integration - we need more than that, we need the source code of the Arquillian integration to ensure that they haven't implemented any smoke and mirrors in their integration. For example, what's to stop them from providing Arquillian integration that actually integrates to a different vendors app server instead of theirs, in that case, the TCK might pass, but only because it was testing a different app server.

So I think any implementation that wishes to claim compatibility must, in addition to publishing test results, also publish any integration code necessary to run the TCK as open source. I would go as far as to say that they should also provide scripts and detailed instructions saying how to run the TCK, since for some vendors app servers this might not be at all trivial.

James,

Thanks for this. I now understand what you're trying to say, and agree that it is a valid point.

I wonder if anyone from Oracle could chime in here and comment on how this is handled currently with Java EE certification?

--
Mike Milinkovich
mike.milinkovich@xxxxxxxxxxxxxxxxxxxxxx
(m) +1.613.220.3223




Avast
                              logo

This email has been checked for viruses by Avast antivirus software.
www.avast.com




_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg

_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg


--
James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper



_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg


Back to the top