Re: [cdt-dev] proposal for side-stepping known test failures

[Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx>]

Thanks, Markus. I’ll update my workspace on Monday and run the tests. Hopefully I’ll see what you’re seeing and remove the exclusions.

 

I suspect the same issue is causing the Codan failures. Can you please take a look at https://bugs.eclipse.org/bugs/show_bug.cgi?id=388764 and see if that’s indeed the case?

 

John

 

 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Schorn, Markus
Sent: Friday, September 14, 2012 2:47 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

 

Hi John,

Recently I did run into the frequent intermittent failures of tests, too. I was able to find the cause, such that both of org.eclipse.cdt.core.suite.AutomatedIntegrationSuite and org.eclipse.cdt.ui.tests.AutomatedSuite work reliably again (for me on Windows) with the exception of two test-cases related to project descriptions.

 

I encourage you to remove the exclusion of the following suites:

     ParserTestSuite, PDOMTests, IndexTests, CallHierarchyTestSuite, SelectionTestSuite,

     RefactoringTestSuite

 

If you do encounter further intermittent failures within these tests, I’ll be happy to look for the cause.

 

FYI, the cause of the issue:

Many test cases need to wait until a file is indexed. There are 2 conditions to check:

·         Indexing for the project is not postponed,

·         Indexer is idle.

I simply made sure all test-cases are using BaseTestCase.waitForIndexer(…), which checks both conditions.

 

Markus.

 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Cortell John-RAT042
Sent: Friday, August 17, 2012 2:18 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

 

I run the tests with high memory ceilings, too. The only difference I see is OS.

 

Anyway, the point is there are failures. The failures are not in code I'm sufficiently familiar with, otherwise I'd troubleshoot them. So, I'm going to make them avoidable as per my proposal, unless there are objections.

 

John

 

---

From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Sergey Prigogin [eclipse.sprigogin@xxxxxxxxx]
Sent: Thursday, August 16, 2012 5:57 PM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

 

On Thu, Aug 16, 2012 at 3:36 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx> wrote:

There are failures in the following plugins when I run the tests locally on my Windows machine. I have the very latest master sources.

 

codan/org.eclipse.cdt.codan.core.test

core/org.eclipse.cdt.core.tests

core/org.eclipse.cdt.ui.tests

 

These tests work for me on Linux when I run them with -Xmx768m -XX:MaxPermSize=192M

 

xlc/org.eclipse.cdt.core.lrparser.xlc.tests (*)

 

These tests were excluded because they were not properly maintained.

 

build/org.eclipse.cdt.make.core.tests (*)

 

Don't know about these. 

 

I’ll be happy to communicate the details to anyone who’s interested in pursuing them. Note that the last two plugins are not in the Hudson report. Do you know why they’re not being tested?

 

Any test that intermittently fails should be optionally avoidable.

 

John

 

-sergey 

 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Sergey Prigogin
Sent: Thursday, August 16, 2012 5:23 PM
To: CDT General developers list.
Subject: Re: [cdt-dev] proposal for side-stepping known test failures

 

https://hudson.eclipse.org/hudson/job/cdt-nightly/lastCompletedBuild/testReport/ shows only two failures, both of which are intermittent. Which test failures are you referring to?

 

-sergey

On Thu, Aug 16, 2012 at 3:03 PM, Cortell John-RAT042 <RAT042@xxxxxxxxxxxxx> wrote:

The CDT tests appear to have known failures--more than just a handful. This makes it very difficult to detect new failures caused by new changes.

 

I’d like to wrap known failures as follows:

 

                if (System.getProperty("cdt.skip.known.test.failures") == null) {

                                // the failing test

                }

 

This will allow me (and others) to get a clean test run before starting on a set of changes. I can then run the tests again after my changes and immediately find out if I’ve broken anything. Right now, I’d have to manually/visually filter out dozens of known failures from the results, which is very tedious, time consuming and error prone. This is much better than simply commenting out broken tests and putting a “TODO”, as it actually leaves the broken tests active in the codebase.

 

Objections?

John

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

 

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev