Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Improved reliability of Index tests

Hi Lukas,

Thank you for the information. I've had intermittent failures on my
fastest PC before so I'll try your scenario a bit later to see if I can
reproduce it. I suggested increasing the timeout because on VERY slow
machines I've seen it go above 10 secs but given that it happens on fast
pcs then it must be a different issue.

> Calling the refresh alone does not improve the situation as it does
> with the workspace builds.
>
> File contents in Eclipse get cached. Since these caches might only be
> updated with resource changes I can see a good chance that rebuilding
> effectively enforces some update of contents which will then get into
> the indexer. 

I'm not clear on which cache might be still be outdated after a refresh
but perhaps that's the case. Hopefully reproducing it on my end can
narrow it down a bit.

Marc-Andre

On 14-03-31 01:21 PM, Lukas Felber wrote:
> Hi Marc-Andre
>
> The effect of failing tests is completely random. The correlation I
> see is, that the more the indexer is used, the more it happens and
> also that if it runs on slower pc's, the failure go away (on our
> one-core vm-build-server test do never fail, on my quad-core pc they
> do a lot). Here at the HSR, I'm also seeing many others with the same
> problems (lots of student projects, mostly cdt-refactoring stuff often
> heavily making use of the indexer) at least until I show them what you
> see now in my gerrit patch.
>
> Basically any test involving the indexer/bindings fails sometimes.
> There is no real pattern its just completely random. To me its pretty
> obvious that cdt-indexing has a synchronization issue.
>
> Failing happens not in waitForIndexer or waitUntilFileIsIndexed but
> rather while asserting results produced by the indexer which are just
> wrong. Meaning I see stuff like "expected:<2> but was:<1>" meaning the
> indexer did not yield 2 but wrongly only one IIndexName.
>
> I can understand your theory about the doing some random stuff might
> make enough delay to help. But since test execution (i did many runs
> of the full cdt-core-tests suite and tonns of 100s of executions of
> IndexBugsTests) do not take longer (difference is marginal) I do not
> think that your theory is right. Also, the removing of Thread.sleep
> calls in a loop speaks for my solution.
>
> Calling the refresh alone does not improve the situation as it does
> with the workspace builds.
>
> File contents in Eclipse get cached. Since these caches might only be
> updated with resource changes I can see a good chance that rebuilding
> effectively enforces some update of contents which will then get into
> the indexer.
>
> Here some stats (average over several 650 IndexBugsTest runs):
>  * Original:                  Failures 6-7, Time 72-81sec
>  * With -Dindexer.timeout=60: Failures 5-7, Time 75-81sec
>  * My solution:               Failures 0-0, Time 76-80sec
>
> Can you reproduce anything like this on your computer?
>
> Basically, I also must say if adding/increasing some timeout somewhere
> helps, then the solution is broken. To fix such issues, some
> synchronisation/joining is missing. And thats the point where a fix
> should come from. I hope that you see it that way too!
>
> Lukas
>
>
> On 31.03.2014 16:42, Marc-André Laperle wrote:
>> Hi Lukas,
>>
>> I've seen tests fail before because of waitForIndexer timing out.
>> Which tests are failing for you? Do they fail with an asserting
>> failed in waitForIndexer or waitUntilFileIsIndexed? I think the
>> reason why adding the workspace.build calls might be helping is that
>> it adds an additional delay and gives the indexer a better chance to
>> finish. But this is just a theory. I didn't think the index tests
>> depended on the build and if they do, I'd be curious to know why.
>> Perhaps it's only the refresh that's necessary? Then I wonder why
>> that would be the case because I believe all the files are created in
>> the workspace through the eclipse.resources APIs.
>>
>> Could you try increasing the indexer timeout and see if that also
>> improves for you? You can add -Dindexer.timeout=60 in your VM
>> arguments (that's if you launch from inside Eclipse, for maven you
>> have to add it in the pom.xml files).
>>
>> Marc-Andre
>> ________________________________________
>> From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on
>> behalf of Lukas Felber [lfelber@xxxxxx]
>> Sent: Monday, 31 March 2014 8:08 AM
>> To: CDT General developers list.
>> Subject: Re: [cdt-dev] Improved reliability of Index tests
>>
>> Hi Marc-André
>>
>> Please see https://git.eclipse.org/r/#/c/24080/ again. It is now rebased
>> to master. The new build even got +1 from hudson.
>> I actually do not exactly know at what point the
>> workspace.build(IncrementalProjectBuilder.FULL_BUILD) method helps the
>> cdt index tests. It seems to refresh resources and hence get rid of
>> outdated cache stuff. This then makes the index more up to date than
>> without rebuilding.
>>
>> Lukas
>>
>>
>> On 28.03.2014 17:59, Marc-André Laperle wrote:
>>> Hi Lukas,
>>>
>>> Can you explain what is the purpose of the build? Also, if you rebase
>>> your patch, I made changed to CDescriptorTests and codan so that they
>>> run more reliably.
>>>
>>> Marc-André
>>>
>>> On 14-03-28 12:58 PM, Lukas Felber wrote:
>>>> Hello Everyone
>>>>
>>>> I created a small patch ( https://git.eclipse.org/r/24080 ) that
>>>> improves the reliability of index tests.
>>>> When running the 65 IndexBugsTests 50 times (total 3250 test runs)
>>>> from a test suite, I got 90 to 100 random test failures.
>>>> After my adaptions, failures are reduced to 0-5 failures (sadly not to
>>>> 0). Meaning a failure reduction of ~95%. When running all core tests,
>>>> I never saw an index test fail (in 10+ runs). Those CDescriptorTests
>>>> do still fail (I did not touch these).
>>>> The adaptions do not have any impact on run-time (+/- 3%). And I was
>>>> able to remove the very ugly part that called Thread.sleep() in a
>>>> loop.
>>>> The better result is achieved by having two additional calls to
>>>> workspace.build(IncrementalProjectBuilder.FULL_BUILD) which basically
>>>> is similar to a "clean..." call from the "Project" eclipse ui-menu.
>>>>
>>>> I hope that some of you guys can try the patch and check if it does
>>>> not have any negative effects somewhere (I only checked the cdt-core
>>>> and codan tests).
>>>>
>>>> Cheers
>>>> Lukas
>>>>
>>>> PS: I will most likely not be active in CDT development in the near
>>>> future since I'm leaving HSR and moving on to IBM.
>>>> _______________________________________________
>>>> cdt-dev mailing list
>>>> cdt-dev@xxxxxxxxxxx
>>>> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>>>
>>> _______________________________________________
>>> cdt-dev mailing list
>>> cdt-dev@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>>>
>> _______________________________________________
>> cdt-dev mailing list
>> cdt-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>> _______________________________________________
>> cdt-dev mailing list
>> cdt-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cdt-dev



Back to the top