| Re: [cdt-dev] proposal for side-stepping known test failures |
|
Andrew,
If the filters I added did not cover failures seen by someone else, and that person also wants to have a reliable test suite, then he should add additional filters. If I see additional unreliable tests in future runs, I will add filters for them.
I wish creating a bugzilla report would result in these intermittent failures being fixed. But I'm not going to hold my breath and I'm not going to waste time creating bugzilla reports for tests that someone has written but has not ensured they run successfully
and consistently on Windows. As you already know, there are intermittent failures even on Linux. I've spent a considerable amount of time debugging and fixing ones I could tackle. But I have to move on.
What I want is to be able to run a CDT test suite locally and have zero failures, based on what's in git. Sadly, with these filters, I've reduced the overall test suite from about 12K tests to 3K[*]. I've had to filter entire chunks because failures happen
not only intermittently but in random locations within the particular chunk. Hopefully someone who has the time and motivation to run and troubleshoot these tests on Windows will try to address this situation.
I said it before and I'll say it again: a test suite that produces dozens of failures is all but useless for regression detection. I want a test suite that passes 100%. What I've done is rigged the tests so that I and others can get that subset with the
flip of a switch.
John
[*] Again, these filters must be manually activated. By default, all tests still run.
From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Andrew Gvozdev [angvoz.dev@xxxxxxxxx]
Sent: Tuesday, August 21, 2012 1:31 PM To: CDT General developers list. Subject: Re: [cdt-dev] proposal for side-stepping known test failures Hi John,
This approach does not look particularly scalable. Is it intended that every commiter will add to the CDT code filters for individual tests that fail in his/her environment?
// Consistently fails, and, yes, I do have Cygwin in my PATH
if (System.getProperty("cdt.skip.known.test.failures") != null) { //$NON-NLS-1$
return;
}
Why don't create a bug in bugzilla instead to fix the problem for good?
Thanks,
Andrew
On Fri, Aug 17, 2012 at 12:49 PM, Cortell John-RAT042
<RAT042@xxxxxxxxxxxxx> wrote:
|