Comment 1 Andrey Loskutov 2015-05-06 16:40:14 EDT

See also bug 448832 comment 8

Comment 2 Lars Vogel 2015-05-06 16:43:11 EDT

FYI - Mac tests seems to be relatively unstable, at least in the past. IIRC I saw several bugs flying by, in which Markus Keller requested an restart of the Mac Box or installation of updates on it. By this instability, I learned to ignore tests which are only failing on Mac.

Comment 3 Andrey Loskutov 2015-05-06 16:58:06 EDT

(In reply to Lars Vogel from comment #2)
> FYI - Mac tests seems to be relatively unstable, at least in the past.
The tests are indeed unstable. They fail in I20150501-2000, but not in I20150503-2000. Next fail in I20150504-2000 and I20150505-2000.

Should we add if(isOnMac()) assume ... ?

Comment 4 Lars Vogel 2015-05-06 17:06:26 EDT

(In reply to Andrey Loskutov from comment #3)
> (In reply to Lars Vogel from comment #2)
> > FYI - Mac tests seems to be relatively unstable, at least in the past.
> The tests are indeed unstable. They fail in I20150501-2000, but not in
> I20150503-2000. Next fail in I20150504-2000 and I20150505-2000.
> 
> Should we add if(isOnMac()) assume ... ?

At least I do that. As I have no Mac and the test failures on Mac seem to follow a random walk pattern. 

Brian, Markus (which AFAIK use a Mac) or other Mac users might see that different. ;)

Comment 5 Lars Vogel 2015-05-15 04:02:57 EDT

Random walk for Mac tests continues, not failing anymore. http://download.eclipse.org/eclipse/downloads/drops4/I20150514-2000/testResults.php

Comment 6 Lars Vogel 2015-05-15 04:03:32 EDT

Verified by looking at http://download.eclipse.org/eclipse/downloads/drops4/I20150514-2000/testResults.php

Comment 7 Andrey Loskutov 2015-05-15 04:12:37 EDT

(In reply to Lars Vogel from comment #5)
> Random walk for Mac tests continues, not failing anymore.
> http://download.eclipse.org/eclipse/downloads/drops4/I20150514-2000/
> testResults.php

I can be wrong but it could be that this was a side effect of https://git.eclipse.org/r/47567/
This closed ~70 dialog instances hanging around after wizard tests. Can be that those opened dialogs affected Mac tests too.

Comment 8 Dani Megert 2015-05-16 03:43:52 EDT

http://download.eclipse.org/eclipse/downloads/drops4/I20150515-2000/testresults/html/org.eclipse.e4.ui.tests_macosx.cocoa.x86_64_7.0.html

Comment 9 Dani Megert 2015-05-21 04:44:36 EDT

The two tests failed again:
http://download.eclipse.org/eclipse/downloads/drops4/I20150520-2000/testresults/html/org.eclipse.e4.ui.tests_macosx.cocoa.x86_64_7.0.html

Comment 10 Markus Keller 2015-05-27 10:23:38 EDT

On Mac and Windows, I constantly see a single test failure when running PartRenderingEngineTests with application "[No Application] - Headless Mode":

java.lang.NullPointerException
	at org.eclipse.e4.ui.tests.workbench.PartRenderingEngineTests.testBug374326(PartRenderingEngineTests.java:3288)

org.eclipse.e4.ui.tests.UIAllTests is always green for me (Mac and Windows) after fixing the invalid shared launch configuration that still tries to run J2SE-1.5.

When I run PartRenderingEngineTests on the Mac and repeatedly click the host workbench window to keep the focus away from the test windows, I can reproduce these failures in testBug326175_False and testBug324839 (plus similar failures in testBug372226 and testBug374326).

I've disabled the failing tests on Mac for now: http://git.eclipse.org/c/platform/eclipse.platform.ui.git/commit/?id=202cc31bfec675bb46670d2b5ca7e9dec9c99e0c

Comment 11 Lars Vogel 2015-05-27 10:32:16 EDT

(In reply to Markus Keller from comment #10)
> On Mac and Windows, I constantly see a single test failure when running
> PartRenderingEngineTests with application "[No Application] - Headless Mode":
> 
> java.lang.NullPointerException
> 	at
> org.eclipse.e4.ui.tests.workbench.PartRenderingEngineTests.
> testBug374326(PartRenderingEngineTests.java:3288)

I'm low on details here but IIRC the tests need to run in a certain order to work because some initialization is necessary. Paul may remember the details IIRC he fixed that a few years ago.

Comment 12 Markus Keller 2015-05-27 10:56:27 EDT

(In reply to Lars Vogel from comment #11)
PartRenderingEngineTests#testBug374326() also fails with an NPE when I run only that test. And it doesn't fail when I run a variant of UIAllTests that just adds that test but not the whole PartRenderingEngineTests.
=> The necessary initialization must come from another test class. It's not (only) about the order of tests in PartRenderingEngineTests.

Paul also didn't seem to know why the test is failing, see bug 383569 comment 28.

Comment 13 Dani Megert 2015-08-30 02:43:20 EDT

testBug372226 now fails constantly.

Comment 14 Markus Keller 2015-10-29 13:03:08 EDT

(In reply to Dani Megert from comment #13)
> testBug372226 now fails constantly.

It doesn't always fail, but it fails more often. E.g. in I20151028-0800, it passed, but in I20151028-2000, it failed again. Screenshots look identical.

Comment 15 Dani Megert 2016-02-04 07:56:30 EST

Failed again in http://download.eclipse.org/eclipse/downloads/drops4/M20160203-1000/testresults/html/org.eclipse.e4.ui.tests_macosx.cocoa.x86_64_7.0.html

Comment 16 Markus Keller 2016-02-09 12:45:22 EST

The intermittent focus issues on the Mac are probably due to parallel test executions on mac-tests2. See bug 487436 comment 5 ff.

Comment 17 Dani Megert 2016-02-11 01:56:16 EST

(In reply to Markus Keller from comment #16)
> The intermittent focus issues on the Mac are probably due to parallel test
> executions on mac-tests2. See bug 487436 comment 5 ff.

That didn't help. testBug372226 failed again in N20160210-2000:

Application does not have an active window

java.lang.IllegalStateException: Application does not have an active window
at org.eclipse.e4.ui.internal.workbench.ApplicationPartServiceImpl.getActiveWindowService(ApplicationPartServiceImpl.java:39)
at org.eclipse.e4.ui.internal.workbench.ApplicationPartServiceImpl.activate(ApplicationPartServiceImpl.java:80)
at org.eclipse.e4.ui.tests.workbench.PartRenderingEngineTests.testBug372226(PartRenderingEngineTests.java:3193)
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:743)
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:351)
at org.eclipse.test.CoreTestApplication.runTests(CoreTestApplication.java:36)
at org.eclipse.test.CoreTestApplication.run(CoreTestApplication.java:32)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:670)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:609)
at org.eclipse.equinox.launcher.Main.run(Main.java:1516)
at org.eclipse.equinox.launcher.Main.main(Main.java:1489)
at org.eclipse.core.launcher.Main.main(Main.java:34)

Comment 18 Brian de Alwis 2016-02-11 09:05:55 EST

I'm baffled by these errors.  The only time I've ever seen them locally is if I've had another application come to the foreground (like AntiRSI.app, which forces me to take breaks).

I noticed JDT has some auto-screenshotting functions in their tests.  Does that require any special setup — is it perhaps a separate JUnit runner?

Comment 19 David Williams 2016-02-16 23:29:41 EST

(In reply to Brian de Alwis from comment #18)
> I'm baffled by these errors.  The only time I've ever seen them locally is
> if I've had another application come to the foreground (like AntiRSI.app,
> which forces me to take breaks).
> 
> I noticed JDT has some auto-screenshotting functions in their tests.  Does
> that require any special setup — is it perhaps a separate JUnit runner?

I am not sure how JDT does it. But I know in the test framework we do take 2 screen shots if a test suite is about to reach the timeout limit. There we start up a separate process/thread. I am not sure how you would want to "work this in" to your tests, but you might be able to call something "direct" in 'startup' or the problematic unit tests and avoid threading and timing complications? 

In the test framework, we use "AWT to take the screen shots, since in that case swt may be 'hung' and would not be able to take a screen shot. 

The code is in the eclipse.platform.releng Git repository. If you import as projects, it is in the org.eclipse.test project. One class is AwtScreenshot. 
The method that does most of the work is dumpAwtScreenshot(String). 

I'd suggest if you want to use this same type of thing, that you just copy/paste the code to your test suite. 

HTH

Comment 20 Dani Megert 2016-02-17 05:13:55 EST

(In reply to Brian de Alwis from comment #18)
> I'm baffled by these errors.  The only time I've ever seen them locally is
> if I've had another application come to the foreground (like AntiRSI.app,
> which forces me to take breaks).
> 
> I noticed JDT has some auto-screenshotting functions in their tests.  Does
> that require any special setup — is it perhaps a separate JUnit runner?

Take a look at org.eclipse.ui.workbench.texteditor.tests.ScreenshotTest


> In the test framework, we use "AWT to take the screen shots, since in that 
> case swt may be 'hung' and would not be able to take a screen shot. 

See org.eclipse.test.AwtScreenshot

Comment 21 Eclipse Genie 2016-02-23 11:47:22 EST

New Gerrit change created: https://git.eclipse.org/r/67161

Comment 22 Eclipse Genie 2016-02-23 13:53:46 EST

Gerrit change https://git.eclipse.org/r/67161 was merged to [master].
Commit: http://git.eclipse.org/c/platform/eclipse.platform.ui.git/commit/?id=98d15fafeb250179cc42b93054cebe49ee5c7d3e

Comment 23 Brian de Alwis 2016-02-23 13:56:13 EST

I pushed a change to ensure the PartRenderingEngineTests have a Realm and UISynchronizer in their context.  @UIEventTopic methods aren't fired if there is no UISynchronizer.

Comment 24 Brian de Alwis 2016-02-25 09:54:27 EST

But that's not enough: http://download.eclipse.org/eclipse/downloads/drops4/N20160224-2000/testresults/html/org.eclipse.e4.ui.tests_macosx.cocoa.x86_64_8.0.html

Comment 25 Lars Vogel 2016-04-25 15:11:34 EDT

Mass move to 4.6 RC1. We might push out more to 4.7.

Comment 26 Dani Megert 2016-11-10 12:50:07 EST

testBug372226 failed again in our last M-build:
http://download.eclipse.org/eclipse/downloads/drops4/M20161110-0200/testresults/html/org.eclipse.e4.ui.tests_ep46M-unit-mac64_macosx.cocoa.x86_64_8.0.html

Maybe we have to disable that one too.

Comment 27 Dani Megert 2017-05-25 05:48:12 EDT

They are constantly failing. Can someone take a look?

Lakshmi, maybe you can spend an hour on them?

Comment 28 Lakshmi P Shanmugam 2017-06-01 03:11:06 EDT

I removed the check (if (checkMacBug466636()) in tests - testBug326175_False and testBug324839 and ran UIAllTests locally, both the tests passed. testBug372226 also passed locally.
I'm unable to run only PartRenderingEngineTests successfully, seems to hang during the first test.
Will continue investigation for 4.8.

Comment 29 Dani Megert 2017-07-19 06:19:25 EDT

Failed again: http://download.eclipse.org/eclipse/downloads/drops4/I20170718-0355/testresults/html/org.eclipse.e4.ui.tests_ep48I-unit-mac64_macosx.cocoa.x86_64_8.0.html


Maybe related to bug 517231?

Comment 30 Lakshmi P Shanmugam 2017-07-20 08:32:48 EDT

(In reply to Dani Megert from comment #29)
> Failed again:
> http://download.eclipse.org/eclipse/downloads/drops4/I20170718-0355/testresults/html/org.eclipse.e4.ui.tests_ep48I-unit-mac64_macosx.cocoa.x86_64_8.0.html
> 
> 
> 
> Maybe related to bug 517231?
I don't think they are directly related. But in this case too may be there is some other active window open causing the test to fail with 'Application does not have an active window'.
I think we should capture a screenshot and see if there are other active dialogs/windows when the test fails. Will provide a patch.

Comment 31 Eclipse Genie 2017-07-21 07:59:16 EDT

New Gerrit change created: https://git.eclipse.org/r/101724

Comment 32 Lakshmi P Shanmugam 2017-07-21 08:02:12 EDT

(In reply to Eclipse Genie from comment #31)
> New Gerrit change created: https://git.eclipse.org/r/101724

The patch uses Screenshots API from org.eclipse.test.performance bundle. The API fails to take screenshot when used with Platform UI tests, opened Bug 520012 & submitted patch to fix it.

Comment 33 Eclipse Genie 2017-07-22 11:47:14 EDT

Gerrit change https://git.eclipse.org/r/101724 was merged to [master].
Commit: http://git.eclipse.org/c/platform/eclipse.platform.ui.git/commit/?id=07abe0e2b4970c83ae2726ef6342d36aa0dc76cd

Comment 34 Dani Megert 2017-09-13 05:41:52 EDT

Now that we capture screenshots we should re-enable the disabled tests in M3 to get more data.

Comment 35 Dani Megert 2017-11-01 11:34:58 EDT

(In reply to Dani Megert from comment #34)
> Now that we capture screenshots we should re-enable the disabled tests in M3
> to get more data.

Ping!

Comment 36 Eclipse Genie 2017-11-03 07:55:21 EDT

New Gerrit change created: https://git.eclipse.org/r/110962

Comment 37 Eclipse Genie 2017-11-03 07:55:22 EDT

New Gerrit change created: https://git.eclipse.org/r/110962

Comment 38 Eclipse Genie 2017-11-08 06:59:59 EST

Gerrit change https://git.eclipse.org/r/110962 was merged to [master].
Commit: http://git.eclipse.org/c/platform/eclipse.platform.ui.git/commit/?id=dbfe95621a35f65a93aaf99ee714ca5d4d13b81b

Comment 39 Mickael Istria 2017-11-29 11:37:26 EST

Is there anything left to do on this topic? If no, please mark it as resolved; otherwise, please retarget to a future milestone.

Comment 40 Dani Megert 2017-12-05 12:10:08 EST

This bug did not get delivered for the specified target milestone. Please set the target milestone when you plan to deliver the fix.

Comment 41 Lakshmi P Shanmugam 2017-12-07 00:35:25 EST

(In reply to Dani Megert from comment #40)
> This bug did not get delivered for the specified target milestone. Please
> set the target milestone when you plan to deliver the fix.

Hi Dani, the tests testBug326175_False and testBug324839 were enabled a month ago (comment#38) and they have not failed since then. The idea was to get more data from the screenshots when the tests fail.
May be we can close the bug for now and reopen if any of them fail?

Comment 42 Lars Vogel 2018-12-06 10:52:38 EST

(In reply to Lakshmi Shanmugam from comment #41)
> (In reply to Dani Megert from comment #40)
> > This bug did not get delivered for the specified target milestone. Please
> > set the target milestone when you plan to deliver the fix.
> 
> Hi Dani, the tests testBug326175_False and testBug324839 were enabled a
> month ago (comment#38) and they have not failed since then. The idea was to
> get more data from the screenshots when the tests fail.
> May be we can close the bug for now and reopen if any of them fail?

+1

Comment 43 Dani Megert 2018-12-10 12:18:04 EST

(In reply to Lars Vogel from comment #42)
> (In reply to Lakshmi Shanmugam from comment #41)
> > (In reply to Dani Megert from comment #40)
> > > This bug did not get delivered for the specified target milestone. Please
> > > set the target milestone when you plan to deliver the fix.
> > 
> > Hi Dani, the tests testBug326175_False and testBug324839 were enabled a
> > month ago (comment#38) and they have not failed since then. The idea was to
> > get more data from the screenshots when the tests fail.
> > May be we can close the bug for now and reopen if any of them fail?
> 
> +1

Regarding your "+1". Did you check bugzilla whether the tests did not fail again?

Comment 44 Eclipse Genie 2020-11-30 06:56:50 EST

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're closing this bug.

If you have further information on the current state of the bug, please add it and reopen this bug. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.