Bug 500757 - RCPTT test failures cause product build to fail
Summary: RCPTT test failures cause product build to fail
Status: CLOSED FIXED
Alias: None
Product: Papyrus-rt
Classification: Modeling
Component: releng (show other bugs)
Version: 0.8.0   Edit
Hardware: All All
: P3 major
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL: https://hudson.eclipse.org/papyrus-rt...
Whiteboard:
Keywords: test
Depends on:
Blocks:
 
Reported: 2016-09-02 11:02 EDT by Christian Damus CLA
Modified: 2017-01-31 16:41 EST (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Damus CLA 2016-09-02 11:02:03 EDT
The latest Papyrus-RT-Product build on the master branch has failed because of an RCPTT test failure.  This should not fail the build, but make it unstable, as is the case with JUnit tests.  The problem with failing the build is that then the results are not archived to the 'last successful build' and testing and distribution are blocked.

I think this specific test failure is similar to a JUnit test failure that I fixed recently in which changes to capsule-part mask label CSS resulted in persisted port locations in diagrams becoming invalid (or sufficiently more invalid than previously; the diagram notation has changed a lot)  such that resizing capsule-parts produces odd results.

But, nonetheless RCPTT test failures should yellow-ball the build, not red-ball it.
Comment 1 Eclipse Genie CLA 2016-09-02 11:44:14 EDT
New Gerrit change created: https://git.eclipse.org/r/80313
Comment 3 Christian Damus CLA 2016-09-02 12:38:15 EDT
(In reply to Eclipse Genie from comment #2)
> Gerrit change https://git.eclipse.org/r/80313 was merged to [master].

This should fix the test failure.  But the problem of tests failing the maven build remains.
Comment 4 Peter Cigehn CLA 2016-09-06 07:54:19 EDT
I am not sure if I can interpret the logs correctly, but it looks like we have another case of a failing build due to that the RCPTT tests fails, and thus no build artifacts are produced, and further testing/verification/feedback of the fixed Bugzilla is blocked. In this specific case I was planning on testing the  Gerrit change for Bug 479350, which finally has been merged after being stuck in Gerrit for a very long time. But now I am unable to do so.
Comment 5 Christian Damus CLA 2016-09-06 08:35:25 EDT
(In reply to Peter Cigehn from comment #4)
> testing/verification/feedback of the fixed Bugzilla is blocked. In this
> specific case I was planning on testing the  Gerrit change for Bug 479350,
> which finally has been merged after being stuck in Gerrit for a very long
> time. But now I am unable to do so.

Yes, because now that this bug is merged, the label in the diagram for a newly created capsule-part no longer shows the [1] multiplicity because now the capsule-part is created with "None (1)" replication and, therefore, no lowerValue nor upperValue.

I can fix the RCPTT test case again to get the build going, but still, it just raises the urgency of the build situation.
Comment 6 Christian Damus CLA 2016-09-06 08:58:58 EDT
It appears that there's another test for ports that now cannot find some UI elements because the property-sheet UI for ports is changed (which it should be).  This test needs updating, probably by the developer of bug 479350 who would be in a better position to do so.
Comment 7 Peter Cigehn CLA 2016-09-06 09:05:36 EDT
I think this has been discussed before, but what are the plans to include running the RCPTT-tests already for the Gerrit change? It feels a bit strange that we do not detect failing RCPTT-tests until the Gerrit change has been merged to master. Shouldn't that be checked already when testing the Gerrit change (and give it a -1 in case the RCPTT-test fails).
Comment 8 Christian Damus CLA 2016-09-06 09:40:48 EDT
(In reply to Peter Cigehn from comment #7)
> I think this has been discussed before, but what are the plans to include
> running the RCPTT-tests already for the Gerrit change? It feels a bit
> strange that we do not detect failing RCPTT-tests until the Gerrit change
> has been merged to master. Shouldn't that be checked already when testing
> the Gerrit change (and give it a -1 in case the RCPTT-test fails).

Maybe there's concern about the running time of the Gerrit test builds if the RCPTT is included?  Besides that I think it took considerably longer to get RCPTT running on Neon than the rest of the build, so it wasn't feasible to include it in the Gerrit test builds.

Now that RCPTT seems to be working, perhaps your suggestion is the best solution.
Comment 9 Christian Damus CLA 2016-09-06 10:46:38 EDT
https://git.eclipse.org/r/#/c/80465/ fixes the latest test failures and a new product build is running.
Comment 10 Christian Damus CLA 2016-09-06 10:52:56 EDT
(In reply to Christian W. Damus from comment #8)
> 
> Now that RCPTT seems to be working, perhaps your suggestion is the best
> solution.

Actually, that's not a solution to this bug.  While it may still be a good suggestion, it is unrelated:  in any case, test failures should only mark a build as unstable, not fail it.  And a Gerrit build job can decide whether to vote +1, 0, or -1 for verification on an unstable build.
Comment 12 Peter Cigehn CLA 2016-11-11 08:36:19 EST
Not sure I feel I have the competence to test and verify this Bugzilla. Unassigning myself as QA.
Comment 13 Charles Rivet CLA 2017-01-31 16:41:03 EST
Batch closing of old, fixed bugs