Bug 385348 - Asynchronous test that never calls back should count as a failure
Summary: Asynchronous test that never calls back should count as a failure
Status: RESOLVED FIXED
Alias: None
Product: Orion (Archived)
Classification: ECD
Component: Client (show other bugs)
Version: 0.5   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 1.0 M1   Edit
Assignee: Mark Macdonald CLA
QA Contact:
URL:
Whiteboard:
Keywords: test
Depends on:
Blocks:
 
Reported: 2012-07-17 16:23 EDT by Mark Macdonald CLA
Modified: 2012-07-18 10:43 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Macdonald CLA 2012-07-17 16:23:05 EDT
Many async tests for Orion components are written such that the test's returned promise resolves iff the test passes, but a test failure does not necessarily reject the promise -- it's just left hanging. 

The test harness ignores these zombie tests (they aren't included in the pass/fail count). This is misleading, as breakage can slip by unnoticed if you're not paying close attention.

To prevent this, the test harness should automatically reject any async test promises who have not have not resolved or rejected after a timeout period.
Comment 1 Mark Macdonald CLA 2012-07-17 18:59:27 EDT
Added a timeout of 8 seconds. I chose this value to exceed the plugin-load timeout (5 seconds), so that the pluginregistry's unit tests for plugin-load timeout would not be declared dead by the test runner and generate spurious failures.

http://git.eclipse.org/c/orion/org.eclipse.orion.client.git/commit/?id=9d8fe690ff557b47f5e696fbfe383d7632bd720e
Comment 2 Simon Kaegi CLA 2012-07-18 09:46:14 EDT
That's great.

I think 8 is cutting it a little close for a default -- suggest 30. Still "infinitely" better than what we have currently.

Might also be an idea to look for a "timeout" property on the test function to override the default for any long running functions.
Comment 3 Mark Macdonald CLA 2012-07-18 10:43:54 EDT
(In reply to comment #2)
> I think 8 is cutting it a little close for a default -- suggest 30.
> 
> Might also be an idea to look for a "timeout" property on the test function to
> override the default for any long running functions.

Good ideas, added here

http://git.eclipse.org/c/orion/org.eclipse.orion.client.git/commit/?id=399f1ffa63d03077bb9c26f954af2a7e46d0e7f3