Bug 574649 - Timeout of test suites is not reported in SDK build
Summary: Timeout of test suites is not reported in SDK build
Status: NEW
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 4.21   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Platform-Releng-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-07-05 04:05 EDT by Andrey Loskutov CLA
Modified: 2021-07-08 02:48 EDT (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Loskutov CLA 2021-07-05 04:05:03 EDT 
Observation from bug 574647. 

ApiToolsPluginTestSuite did not finish properly (run in timeout) but the SDK results did not show any error / crash, the ApiToolsPluginTestSuite entry is just missing.

Compare 

bad: 
https://download.eclipse.org/eclipse/downloads/drops4/I20210704-1800/testresults/html/org.eclipse.pde.api.tools.tests_ep421I-unit-cen64-gtk3-java11_linux.gtk.x86_64_11.html

good:

https://download.eclipse.org/eclipse/downloads/drops4/I20210703-0600/testresults/html/org.eclipse.pde.api.tools.tests_ep421I-unit-cen64-gtk3-java11_linux.gtk.x86_64_11.html

I have no idea where the build results are observed/reported as pass/fail/crash, is this in each test suite or in the main script?
Comment 1 Andrey Loskutov CLA 2021-07-05 04:09:11 EDT 
@Vikas: can you please add 4.21 version for the "Version" field in bugzilla?
Comment 2 Andrey Loskutov CLA 2021-07-05 04:22:11 EDT 
org.eclipse.jdt.core.tests.dom did not finished too in SDK builds for same reason, and that was unnoticed => common issue.
Comment 3 Sravan Kumar Lakkimsetti CLA 2021-07-05 04:57:54 EDT 
Normally the tests return as DNF. In some cases the teams package tests into separate bundles but combining them to a single report. In case of ApiTools There are two separate test bundles updating a single test report. 

At releng framework level we look for whether the test results page exists. If not we specify are DNF. 

In case of ApiTools we do have test results as one of the test bundles succeeded. So this is masking test failures. 

There are 2 options
1. Split the test bundles to generate individual test reports
2. The test result report generator is available at https://git.eclipse.org/c/platform/eclipse.platform.releng.buildtools.git/tree/bundles/org.eclipse.build.tools/src/org/eclipse/releng/generators/TestResultsGenerator.java. Update this one to identify timeouts or errors occured
Comment 4 Vikas Chandra CLA 2021-07-08 02:48:38 EDT 
(In reply to Andrey Loskutov from comment #1)
> @Vikas: can you please add 4.21 version for the "Version" field in bugzilla?

I just saw this notification. I have added version in PDE