[platform-releng-dev] Test environment / isolation?

[Stephan Herrmann <stephan.herrmann@xxxxxxxxx>]

Hi,

In the past we have seen some ways in which mere presence of, e.g.,
org.eclipse.jdt.launching could influence JDT/Core tests.

Today I reproduced a case, where presence of o.e.jdt.debug.tests
causes exceptions in a JDT/Core test (didn't actually fail the
test, but still looks brittle).

This lets me ask: what level of test isolation can we assume for
official builds? Are tests for each component run in an environment
that only contains this component and its dependencies, or are
all components of the SDK installed into one big thing and then
all tests run on top of that?

FYI, the recent failure involved the following chain of events:

- jdt.core:
    Test JavaProjectTests.testBug462756() is run in headless mode

- jdt.launching:
    JREPreferenceModifyListener.preApply() is triggered

- jdt.debug.tests:
    VMInstallTestsLibraryLocationResolver.getSourcePath() is called

- the latter tries to DebugUIPlugin.start()
    which fails because the workbench has not been started (since running headlessly)


Is it OK to assume that such interaction will not occur during builds,
or should we make our tests robust against such?

thanks,
Stephan