[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] proposal for side-stepping known test failures



On Thu, Aug 16, 2012 at 5:18 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx> wrote:
I run the tests with high memory ceilings, too. The only difference I see is OS.

Anyway, the point is there are failures. The failures are not in code I'm sufficiently familiar with, otherwise I'd troubleshoot them. So, I'm going to make them avoidable as per my proposal, unless there are objections.

This is a slippery slope. Please don't consider you testing adequate if you see failures of tests that don't happen on Hudson.Â

John

-sergeyÂ


From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Sergey Prigogin [eclipse.sprigogin@xxxxxxxxx]
Sent: Thursday, August 16, 2012 5:57 PM

To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures



On Thu, Aug 16, 2012 at 3:36 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx> wrote:

There are failures in the following plugins when I run the tests locally on my Windows machine. I have the very latest master sources.

Â

codan/org.eclipse.cdt.codan.core.test

core/org.eclipse.cdt.core.tests

core/org.eclipse.cdt.ui.tests


These tests work for me on Linux when I run them withÂ-Xmx768m -XX:MaxPermSize=192M

xlc/org.eclipse.cdt.core.lrparser.xlc.tests (*)


These tests were excluded because they were not properly maintained.
Â

build/org.eclipse.cdt.make.core.tests (*)


Don't know about these.Â

Â

Iâll be happy to communicate the details to anyone whoâs interested in pursuing them. Note that the last two plugins are not in the Hudson report. Do you know why theyâre not being tested?

Â

Any test that intermittently fails should be optionally avoidable.

Â

John


-sergeyÂ

Â

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Sergey Prigogin
Sent: Thursday, August 16, 2012 5:23 PM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

Â

https://hudson.eclipse.org/hudson/job/cdt-nightly/lastCompletedBuild/testReport/ shows only two failures, both of which are intermittent. Which test failures are you referring to?

Â

-sergey

On Thu, Aug 16, 2012 at 3:03 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx> wrote:

The CDT tests appear to have known failures--more than just a handful. This makes it very difficult to detect new failures caused by new changes.

Â

Iâd like to wrap known failures as follows:

Â

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if (System.getProperty("cdt.skip.known.test.failures") == null) {

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ // the failing test

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ }

Â

This will allow me (and others) to get a clean test run before starting on a set of changes. I can then run the tests again after my changes and immediately find out if Iâve broken anything. Right now, Iâd have to manually/visually filter out dozens of known failures from the results, which is very tedious, time consuming and error prone. This is much better than simply commenting out broken tests and putting a âTODOâ, as it actually leaves the broken tests active in the codebase.

Â

Objections?

John


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

Â


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev