Community
Participate
Working Groups
Tests cannot be launched with a profiling set. Although launching a manual test with a profiling set is not a valid use case, executing the other JUnit-based test types with a profiling set is a very plausible use case. For example, using the Method Code Coverage profiling set to determine API test coverage or using the Execution Time Analysis profiling set to determine performance statistics. Each profiling set provides a valuable benefit to evaluating and analyzing a test's execution. In the Profile launch configuration dialog (see screen capture), a Test launch configuration can be created but all of the tests are filtered from the Test tabbed pane. Once the tests are exposed to the user to executed with a profiling set, the configured profiling set must be propagated to the execution harness and runner for configuring hte test's execution.
Created attachment 39580 [details] Screen capture.
This defect is also required for the new line level code coverage feature (https://bugs.eclipse.org/bugs/show_bug.cgi?id=129095) being released as a Technology Preview in TPTP V4.2.0.
Please consider this a P1 priority since it is required for features schedule for release as a TPTP V4.2.0 Technology Preview.
Per test execution meeting on May 9th, this will be split in two, with the real enhancement work coming after 4.2. The part that remains with this defect is do make sure that the user isn't misled into thinking you can profile the test (solved by not implementing an extension point so tests aren't exposed). The RFE part of this is already covered as part of an existing RFE - 75029. Paul, if you feel uncomfortable with that one, please enter a new RFE specifically. Assigning to Joe to disable the related extension point for test for 4.2 (check with Paul if necessary for more detail).
I am concerned that there may be a consuming product which uses the default "Test" launch configuration type, and does support launching with a Profiling Set. If we disable this UI capability, I think this would cause a regression in that consuming product. Julienn, can you confirm or deny? If there would be no impact, I can certainly disbale this capability. However, if this would cause a regression for downstream products, we'll need to consider other ways to improve this behavior.
(In reply to comment #5) As far as I know, there is (no more) any downstream products that define test types with support for profile mode. However, the extension point org.eclipse.hyades.test.core.launchconfigLaunchableType (which is part of the "API" of TPTP) states that "profile" is a supported mode (provided that the consuming plugin provides the adequate execution components for running its tests in profile mode). If we remove the "Test" launch configuration from the Profile dialog box, we won't be able to honor what the extension point describes, and this would look to me as a defect. (And removing the statement that "profile" is a supported mode would be a regression). On another hand Joe told me that there is a scenario where the user is allowed to launch a test launch configuration in profile mode, even if the test doesn't support profile mode (to do so, create a test launch configuration for a JUnit test, launch it once, then go to the profile dropdown menu and select the first item in the list: the test seems to be launched but fails). Joe, if you assign this bugzilla to me, I'll be happy to fix it.
(In reply to comment #6) Since we are promoting externally to launch tests with profiling sets or data collectors/analysis types (albeit, user-defined through the org.eclipse.hyades.test.core.launchconfigLaunchableType extension point), this defect the original issue of this defect holds. We have to support launching JUnit-based test types with a profiling set or data collectors/analysis types.
I think we all agree that there are two issues here. First, we should allow tests to be profiled. The intent is that this effort is covered by 75029. Beyond that, we have the defect that, today, before 75029 is complete, you are able to launch a test in profile mode by using the profile button when a test launch configuration has already been created. That is a defect, and the error message is misleading. I believe this is the only way that one can launch a test in profile mode, since no test is selectable in profile mode today (and that will be addressed by 75029 when one can be actually profiled.) So, I think Julien's suggestion is the best route for dealing with this in the short term, and when 75029 is done, we will be able to profile tests, so these restrictions will be removed. Paul -- do you agree?
Changing summary to reflect the defect tracked by this bugzilla. The more general issue of not being able to profile a test will be tracked by bugzilla 75029.
(In reply to comment #8) Yes.
Fix checked in for 4.2.1. Because there is no PII drop in 4.2.1, the message issued by TPTP is generic ("Problems found in Launch Configuration. See error log for detailed status."). The error log provides a non-translated informational message ("The launch mode "profile" is not supported for the test specified in the launch configuration"). I am retargetting this defect to 4.3 so that the detailed message can be translated and directly issued to the user without him having to look at the error log.
The fix to org.eclipse.hyades.test.core.launch.delegates.AbstractLaunchConfigurationDelegate2.java causes the following exception when trying to run a test from a self-hosted Eclipse workbench (required for development/debugging): java.lang.Throwable: The launch mode "run" is not supported for the test specified in the launch configuration at org.eclipse.hyades.test.core.TestCorePlugin.logError(TestCorePlugin.java:214) at org.eclipse.hyades.test.core.launch.delegates.AbstractLaunchConfigurationDelegate2.validate(AbstractLaunchConfigurationDelegate2.java:90) at org.eclipse.hyades.test.core.launch.delegates.BasicTestLaunchConfigurationDelegate2.validate(BasicTestLaunchConfigurationDelegate2.java:117) at org.eclipse.hyades.test.core.launch.delegates.AbstractLaunchConfigurationDelegate2.launch(AbstractLaunchConfigurationDelegate2.java:162) at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:639) at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:565) at org.eclipse.debug.internal.ui.DebugUIPlugin.buildAndLaunch(DebugUIPlugin.java:754) at org.eclipse.debug.internal.ui.DebugUIPlugin$6.run(DebugUIPlugin.java:944) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58)
(In reply to comment #12) Fix for this regression commited to CVS.
Completed. (4.3i1)
ACTION: Please verify/close this defect.
Verified on TPTP-4.3.0-200611070100 driver.
To work around this limitation and use the TPTP Method Level Code Coverage function during test execution, do the following: 1) Append the following JVM parameter (PI Agent parameter) to the hyades.test.JVM parameter in the <Agent Controller installation directory>\plugins\org.eclipse.hyades.test.tools.core_<version number>\config\pluginconfig.xml file: <Parameter position="prepend" value="-Dhyades.test.JVM="%JAVA_PATH%" -XrunpiAgent:server=controlled"/> 2) Restart the Agent Controller. 3) Run your test. 4) Attach to the Java Profiler agent in the workbench (Run >> Profile... >> Attach - Java Process). Make sure you set the correct filters and analysis types (e.g. Method Level Code Coverage). Consult the Agent Controller's getting_started.html file for more information on PI Agent parameters. I prefer controlled mode since it stops application until you attach/monitor.
*** Bug 205635 has been marked as a duplicate of this bug. ***
Using 4.4.0.3, when the user attempts to profile a test suite, the following error message is displayed in the Test Launch Configuration dialog: IWAT4029E No Test selected. However, no tests appear to be selected. Reopening to address in 4.5.
Please provide a sizing for this defect before Monday, November 12, 2007.
Deferring to I6.
Needed by consuming product
(In reply to comment #22) > Needed by consuming product > The bast I can parse out of this long bugzilla is that a very important and useful function was for poor reasoning disabled. That is to be able to profile/monitor/instrument the run of a test. Asside from what I think was a mistake, there is now a bug caused by, or part of the "fix" which is now being tracked with this bugzilla. If I have this right, I think there should be a short discussion about the base removal of function, and a separate defect opened to track the bug under consideration at this time.
Deferring to I7 as discussed by PMC/AG (http://wiki.eclipse.org/TPTP-AG-20080331#Test_Project).
(In reply to comment #23) > (In reply to comment #22) > > Needed by consuming product > > > > The bast I can parse out of this long bugzilla is that a very important and > useful function was for poor reasoning disabled. That is to be able to > profile/monitor/instrument the run of a test. Asside from what I think was a > mistake, there is now a bug caused by, or part of the "fix" which is now being > tracked with this bugzilla. Profiling (and debugging) a test execution was never supported, despite allowing test types to be launchable from the run/profile/debug launch type, through the org.eclipse.hyades.test.core.launchconfigLaunchableType extension point. This defect covers changes to improve usability until the support is provided. > If I have this right, I think there should be a short discussion about the base > removal of function, and a separate defect opened to track the bug under > consideration at this time. This support is proposed under enhancement https://bugs.eclipse.org/bugs/show_bug.cgi?id=75029.
(In reply to comment #25) > Profiling (and debugging) a test execution was never supported, despite > allowing test types to be launchable from the run/profile/debug launch type, > through the org.eclipse.hyades.test.core.launchconfigLaunchableType extension > point. > I don't agree and perhaps don't understand what you mean by not supported. It was intentionally possible to do, and was a used capability, just like running the generated code as a standard JUnit. If you simply mean it was not explicitly being tested, that is not a good enough reason to disable valuable function. Not profiling/debugging a manual test is the corner case to handle, not full removal of function. Making test execution aware of this scenario and requiring that integration to do this kind of thing may be interesting but is not the same basic function, and it seems the proposed future release enhancement is going down that path.
Hours worked for investigating the correct solution and creating/testing the patch. After an exhaustive review, here is a summary: -JUnit-based TPTP test suites (e.g. JUnit, JUnit Plug-in, and URL) as well as JUnit test classes may be launched in the Debug launch configuration dialog using the Test launch configuration type. The org.eclipse.hyades.execution.harness.TestExecutionHarnessExecutorStub.java checks the launch mode (e.g. debug) and adds the necessary debug flags to the org.eclipse.hyades.execution.local.JavaProcessExecutableObjectStub, launches, and attaches to the remote VM (see org.eclipse.hyades.execution.harness.TestExecutionHarnessExecutorStub.debugAndLaunch(IProgressMonitor)). If breakpoints are set in the JUnit test classes, the test suspends execution at the first breakpoint until the user continues (steps/runs/etc.) with the debugger. The only caveat is that the Debug perspective is not opened after the launch (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=75029#c15). -Although the org.eclipse.hyades.test.core.launchconfigLaunchableType extension point and the org.eclipse.hyades.test.core.launch.delegates.BasicTestLaunchConfigurationDelegate2.java supports a 'profile' mode, there is no mechanism for the org.eclipse.hyades.execution.harness.TestExecutionHarnessExecutorStub.java or its subclasses to access profile configuration (data collectors/analysis types/files/destination configuration/etc) to propagate to the org.eclipse.hyades.execution.local.JavaProcessExecutableObjectStub. Enhancement 75029 is required before users/extenders/consuming products can launch JUnit-based TPTP test suites (e.g. JUnit, JUnit Plug-in, and URL) as well as JUnit test classes in the Profile launch configuration dialog using the Test launch configuration type. As such, the solution for this defect is to disable the Test launch configuration type until enhancement 75029 is complete.
Created attachment 96650 [details] Patch.
Patch checked-in to CVS (HEAD).
Verified in TPTP-4.5.0-200806070100. Closing.
