Re: [cdt-dev] proposal for side-stepping known test failures

[Marc Khouzam <marc.khouzam@xxxxxxxxxxxx>]

Hi,

although the change is a bit awkward, we must take a step back and think of the benefits.

Having 100% success when running the tests greatly improves the value of the regression tests.
As soon as there are a couple of failures, people tend to dismiss any failure as "must be one
of those expected ones", and new failures go unnoticed.  

BTW, running the test suite before a change and then again after the change to compare the 
results is a big time waster for the designer, so I've never liked that approach.

Finally, since we don't have Hudson running tests on Windows (could we though, as Andrew O
mentions?), we must count on people running them manually.  In that case, we should make their
life as easy as possible, or they will stop doing it.

Using this system property in some places, although awkward, has no real negative impact.  For test code
I think this is acceptable considering the benefits.  I know that in the GDB project (they don't use JUnit), they
have a way to mark a test as an "expected failure".

Finally, using a TestSuite as Markus suggests would be more elegant.  However, I don't know how much time
that would require to do.  If the amount of time is small, I prefer that, but we should not spend too much time
on making tests more elegant when that time could be spent on CDT code, or even fixing the tests.

My 2 cents.

Marc
________________________________________
From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Schorn, Markus [Markus.Schorn@xxxxxxxxxxxxx]
Sent: August 22, 2012 3:28 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

I think using the java-property it is a strange approach to get a customized test suite. It would be more natural to compose a separate TestSuite out of the existing tests.
Markus.

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Cortell John-RAT042
Sent: Tuesday, August 21, 2012 9:47 PM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

I brought this up for discussion last week, before I pushed the changes. One person had concerns; we began a debate and the thread ended without retort to my points. Thus I went ahead with my changes.

That said, I will respect a general consensus. I encourage a few more folks to weigh in so we can get there. Honestly, I'm quire surprised that such a small and innocuous set of changes to test code is drawing such a reaction.

John

________________________________
From: cdt-dev-bounces@xxxxxxxxxxx<mailto:cdt-dev-bounces@xxxxxxxxxxx> [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Andrew Gvozdev [angvoz.dev@xxxxxxxxx]
Sent: Tuesday, August 21, 2012 2:36 PM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures
I agree, +1.

There are other ways to achieve test suite that passes 100%. You can create separate test suite that includes only tests you wish to test. You can keep your changes on local branch as Sergey suggested. You can create a special branch in CDT repository even if there is really need to share. But, please, do not keep it on master branch.

Andrew
On Tue, Aug 21, 2012 at 3:05 PM, Sergey Prigogin <eclipse.sprigogin@xxxxxxxxx<mailto:eclipse.sprigogin@xxxxxxxxx>> wrote:
I propose to revert the commit http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=1bd247ce3d8825b76623166a89b111884ae1b195. John, you can keep the commit on a local branch if you prefer.

-sergey

On Tue, Aug 21, 2012 at 11:55 AM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx<mailto:RAT042@xxxxxxxxxxxxx>> wrote:
Andrew,

If the filters I added did not cover failures seen by someone else, and that person also wants to have a reliable test suite, then he should add additional filters. If I see additional unreliable tests in future runs, I will add filters for them.

I wish creating a bugzilla report would result in these intermittent failures being fixed. But I'm not going to hold my breath and I'm not going to waste time creating bugzilla reports for tests that someone has written but has not ensured they run successfully and consistently on Windows. As you already know, there are intermittent failures even on Linux. I've spent a considerable amount of time debugging and fixing ones I could tackle. But I have to move on.

What I want is to be able to run a CDT test suite locally and have zero failures, based on what's in git. Sadly, with these filters, I've reduced the overall test suite from about 12K tests to 3K[*]. I've had to filter entire chunks because failures happen not only intermittently but in random locations within the particular chunk. Hopefully someone who has the time and motivation to run and troubleshoot these tests on Windows will try to address this situation.

I said it before and I'll say it again: a test suite that produces dozens of failures is all but useless for regression detection. I want a test suite that passes 100%. What I've done is rigged the tests so that I and others can get that subset with the flip of a switch.

John

[*] Again, these filters must be manually activated. By default, all tests still run.

________________________________
From: cdt-dev-bounces@xxxxxxxxxxx<mailto:cdt-dev-bounces@xxxxxxxxxxx> [cdt-dev-bounces@xxxxxxxxxxx<mailto:cdt-dev-bounces@xxxxxxxxxxx>] on behalf of Andrew Gvozdev [angvoz.dev@xxxxxxxxx<mailto:angvoz.dev@xxxxxxxxx>]
Sent: Tuesday, August 21, 2012 1:31 PM

To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

Hi John,
This approach does not look particularly scalable. Is it intended that every commiter will add to the CDT code filters for individual tests that fail in his/her environment?
// Consistently fails, and, yes, I do have Cygwin in my PATH
if (System.getProperty("cdt.skip.known.test.failures") != null) { //$NON-NLS-1$
return;
}

Why don't create a bug in bugzilla instead to fix the problem for good?


Thanks,
Andrew


On Fri, Aug 17, 2012 at 12:49 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx<mailto:RAT042@xxxxxxxxxxxxx>> wrote:
I was able to resolve roughly a dozen intermittent failures caused by a bug in

   org.eclipse.cdt.ui.tests.BaseUITestCase.checkTreeNode(IViewPart, int, String)

The implementation searched the entire workbench for the SWT Tree instead of just the given view.  This would intermittently (but frequently enough) return, e.g., the Outline view’s tree because it has a root element with the same label as that in the Call hierarchy view. It was just a matter of the order in which the controls in the workbench were found, which is undefined and inconsistent.

So, here we have an example of a test being unreliable, unrelated to OS or environment. The test had a bug and you and Hudson were simply getting lucky.

I’ve pushed the fix to master.

John


From: cdt-dev-bounces@xxxxxxxxxxx<mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx<mailto:cdt-dev-bounces@xxxxxxxxxxx>] On Behalf Of Andrew Overholt
Sent: Friday, August 17, 2012 7:44 AM

To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures


> Since we don't have Hudson on Windows

There is a Windows Hudson slave, isn't there?  A job could be set up to run either the full CDT build or just the tests on it.

Andrew

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx<mailto:cdt-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx<mailto:cdt-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx<mailto:cdt-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/cdt-dev