Re: [platform-releng-dev] Test environment / isolation?

["Markus Keller" <markus_keller@xxxxxxxxxx>]

Hi Stephan

> Is it OK to assume that such interaction
will not occur during builds, or should we make our tests robust against
such?

The production test runs should provide
the necessary isolation, but developer workspaces don't.

Good citizens make global contributions
in their test projects generally inactive and only influence the workbench
while their associated tests are running.

In this case, org.eclipse.jdt.debug.tests.core.VMInstallTests#isTesting
looks like that was already implemented, but since the parent class imports
a UI class, this accidentally triggers activation of the DebugUIPlugin.
Could you please file a bug for JDT Debug? This shouldn't be too hard to
fix. Moving the isTesting field into another class should be enough.

Unfortunately, some sub-projects have
let their test projects sprawl uncontrolled. E.g. the Platform UI tests
are so bad, that you shouldn't even run a target workspace while these
!@#$% test projects are open. My workaround for that is to group these
tests in a working set and add a dummy project _Eclipse_UI_Tests_Launchblocker
that least allows me to stop a launch before my target workspace gets damaged.
When "Validate Plug-ins automatically prior to launching" is
enabled in the launch configuration, a MANIFEST.MF like this will trigger
a cancellable dialog:

Manifest-Version:
1.0
Bundle-ManifestVersion:
2
Bundle-Name:
Launchblocker
Bundle-SymbolicName:
_Eclipse_UI_Tests_Launchblocker
Bundle-Version:
1.0.0.qualifier
Eclipse-PlatformFilter:
(osgi.os=dont.launch.with.ui.tests)


Markus




From:      
 "Sravan K Lakkimsetti"
<sravankumarl@xxxxxxxxxx>
To:      
 "Eclipse platform
release engineering list." <platform-releng-dev@xxxxxxxxxxx>
Date:      
 2017-02-06 07:45
Subject:    
   Re: [platform-releng-dev]
Test environment / isolation?
Sent by:    
   platform-releng-dev-bounces@xxxxxxxxxxx

---

Hi Stephan,

For the official build we use the SDK build built during our build process.
So it will contain all the components(excluding test components and their
dependencies). 

During the test for a particular component
1. Install test bundles for that component including dependencies(these
are picked up from the Automated tests bundle available in our download
page)
2. Run the tests defined in test.xml
3. Remove test bundles

This process is repeated for all the components. During the test we have
full SDK and test bundles for the component under test(and dependencies)
are installed. 

Hope this helps

Thanks
Sravan



-----Original Message-----
From: Stephan Herrmann [mailto:stephan.herrmann@xxxxxxxxx]
Sent: Sunday, February 05, 2017 10:59 PM
To: Eclipse platform release engineering list. <platform-releng-dev@xxxxxxxxxxx>
Subject: [platform-releng-dev] Test environment / isolation?

Hi,

In the past we have seen some ways in which mere presence of, e.g., org.eclipse.jdt.launching
could influence JDT/Core tests.

Today I reproduced a case, where presence of o.e.jdt.debug.tests causes
exceptions in a JDT/Core test (didn't actually fail the test, but still
looks brittle).

This lets me ask: what level of test isolation can we assume for official
builds? Are tests for each component run in an environment that only contains
this component and its dependencies, or are all components of the SDK installed
into one big thing and then all tests run on top of that?

FYI, the recent failure involved the following chain of events:

- jdt.core:
Test JavaProjectTests.testBug462756() is run in headless mode

- jdt.launching:
JREPreferenceModifyListener.preApply() is triggered

- jdt.debug.tests:
VMInstallTestsLibraryLocationResolver.getSourcePath() is called

- the latter tries to DebugUIPlugin.start()
which fails because the workbench has not been started (since running headlessly)


Is it OK to assume that such interaction will not occur during builds,
or should we make our tests robust against such?

thanks,
Stephan