Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Improved reliability of Index tests

Hi Marc-Andre

The effect of failing tests is completely random. The correlation I see is, that the more the indexer is used, the more it happens and also that if it runs on slower pc's, the failure go away (on our one-core vm-build-server test do never fail, on my quad-core pc they do a lot). Here at the HSR, I'm also seeing many others with the same problems (lots of student projects, mostly cdt-refactoring stuff often heavily making use of the indexer) at least until I show them what you see now in my gerrit patch.

Basically any test involving the indexer/bindings fails sometimes. There is no real pattern its just completely random. To me its pretty obvious that cdt-indexing has a synchronization issue.

Failing happens not in waitForIndexer or waitUntilFileIsIndexed but rather while asserting results produced by the indexer which are just wrong. Meaning I see stuff like "expected:<2> but was:<1>" meaning the indexer did not yield 2 but wrongly only one IIndexName.

I can understand your theory about the doing some random stuff might make enough delay to help. But since test execution (i did many runs of the full cdt-core-tests suite and tonns of 100s of executions of IndexBugsTests) do not take longer (difference is marginal) I do not think that your theory is right. Also, the removing of Thread.sleep calls in a loop speaks for my solution.

Calling the refresh alone does not improve the situation as it does with the workspace builds.

File contents in Eclipse get cached. Since these caches might only be updated with resource changes I can see a good chance that rebuilding effectively enforces some update of contents which will then get into the indexer.

Here some stats (average over several 650 IndexBugsTest runs):
 * Original:                  Failures 6-7, Time 72-81sec
 * With -Dindexer.timeout=60: Failures 5-7, Time 75-81sec
 * My solution:               Failures 0-0, Time 76-80sec

Can you reproduce anything like this on your computer?

Basically, I also must say if adding/increasing some timeout somewhere helps, then the solution is broken. To fix such issues, some synchronisation/joining is missing. And thats the point where a fix should come from. I hope that you see it that way too!

Lukas


On 31.03.2014 16:42, Marc-André Laperle wrote:
Hi Lukas,

I've seen tests fail before because of waitForIndexer timing out. Which tests are failing for you? Do they fail with an asserting failed in waitForIndexer or waitUntilFileIsIndexed? I think the reason why adding the workspace.build calls might be helping is that it adds an additional delay and gives the indexer a better chance to finish. But this is just a theory. I didn't think the index tests depended on the build and if they do, I'd be curious to know why. Perhaps it's only the refresh that's necessary? Then I wonder why that would be the case because I believe all the files are created in the workspace through the eclipse.resources APIs.

Could you try increasing the indexer timeout and see if that also improves for you? You can add -Dindexer.timeout=60 in your VM arguments (that's if you launch from inside Eclipse, for maven you have to add it in the pom.xml files).

Marc-Andre
________________________________________
From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Lukas Felber [lfelber@xxxxxx]
Sent: Monday, 31 March 2014 8:08 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] Improved reliability of Index tests

Hi Marc-André

Please see https://git.eclipse.org/r/#/c/24080/ again. It is now rebased
to master. The new build even got +1 from hudson.
I actually do not exactly know at what point the
workspace.build(IncrementalProjectBuilder.FULL_BUILD) method helps the
cdt index tests. It seems to refresh resources and hence get rid of
outdated cache stuff. This then makes the index more up to date than
without rebuilding.

Lukas


On 28.03.2014 17:59, Marc-André Laperle wrote:
Hi Lukas,

Can you explain what is the purpose of the build? Also, if you rebase
your patch, I made changed to CDescriptorTests and codan so that they
run more reliably.

Marc-André

On 14-03-28 12:58 PM, Lukas Felber wrote:
Hello Everyone

I created a small patch ( https://git.eclipse.org/r/24080 ) that
improves the reliability of index tests.
When running the 65 IndexBugsTests 50 times (total 3250 test runs)
from a test suite, I got 90 to 100 random test failures.
After my adaptions, failures are reduced to 0-5 failures (sadly not to
0). Meaning a failure reduction of ~95%. When running all core tests,
I never saw an index test fail (in 10+ runs). Those CDescriptorTests
do still fail (I did not touch these).
The adaptions do not have any impact on run-time (+/- 3%). And I was
able to remove the very ugly part that called Thread.sleep() in a loop.
The better result is achieved by having two additional calls to
workspace.build(IncrementalProjectBuilder.FULL_BUILD) which basically
is similar to a "clean..." call from the "Project" eclipse ui-menu.

I hope that some of you guys can try the patch and check if it does
not have any negative effects somewhere (I only checked the cdt-core
and codan tests).

Cheers
Lukas

PS: I will most likely not be active in CDT development in the near
future since I'm leaving HSR and moving on to IBM.
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



Back to the top