Bug 507626 - Debug framework should provide a generic "test report" view
Summary: Debug framework should provide a generic "test report" view
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Debug (show other bugs)
Version: 4.6   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: 4.18 RC1   Edit
Assignee: Victor Rubezhny CLA
QA Contact: Jeff Johnston CLA
URL:
Whiteboard:
Keywords: Documentation, noteworthy, plan
Depends on:
Blocks:
 
Reported: 2016-11-16 12:25 EST by Mickael Istria CLA
Modified: 2021-01-04 11:06 EST (History)
15 users (show)

See Also:


Attachments
First proposal of a unified TestView (380.82 KB, application/zip)
2019-03-11 10:12 EDT, Christian Pontesegger CLA
no flags Details
Old/new view comparison (73.76 KB, image/png)
2019-04-02 17:00 EDT, Christian Pontesegger CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mickael Istria CLA 2016-11-16 12:25:36 EST
Currently, it seems like all test runners to reimplement a new Eclipse view to display test reports. However, test reports have a lot of similarities, whatever runner/language is used.
It would be great for Eclipse Platform to provide a generic view for Test Reports and let the various test launcher hook in it.
Comment 1 Dani Megert CLA 2019-01-29 05:01:18 EST
Do you consider the JUnit view a "test report" view?

There's currently a very related thread on jdt-dev: https://www.eclipse.org/lists/jdt-dev/msg01104.html
Comment 2 Mickael Istria CLA 2019-01-29 05:05:56 EST
(In reply to Dani Megert from comment #1)
> Do you consider the JUnit view a "test report" view?
> 
> There's currently a very related thread on jdt-dev:
> https://www.eclipse.org/lists/jdt-dev/msg01104.html

Yes, that's exactly what I had in mind when opening the bug. I'll link to that bug on the mailing-list.
Comment 3 Christian Pontesegger CLA 2019-01-29 07:55:55 EST
Would like to start to work on that topic as I need a test view for another project. Ideas so far:

1) I would opt for having 1 Unit Test View which could be reused by any language. 

2) I did a rough carve out of the view to a new plugin. A first look at the initial dependencies reveals:
 org.eclipse.debug.core,
 org.eclipse.debug.ui,
 org.eclipse.core.runtime,
 org.eclipse.jdt.junit,
 org.eclipse.jdt.ui,
 org.eclipse.jdt.core,
 org.eclipse.jdt.launching,
 org.eclipse.ui.ide;bundle-version="3.14.300",
 org.eclipse.ui.workbench.texteditor,
 org.eclipse.text,
 org.eclipse.compare,
 org.eclipse.jface.text,
 org.junit,
 org.eclipse.ui.console

Of course we want to get rid of o.e.jdt.*, for the rest I wonder where to put it. My gut feeling would say o.e.debug, but also o.e.platform.ui would be possible. Do all of the remaining dependencies fit to one of these projects?

3) We definitely need a generic model for the unit tests. I already did a version of this in EMF once. I would prefer EMF compared to any hand written code here.

4) For a first implementation I would use as much as possible from existing JDT code by copying relevant classes. Then we could refactor it and add generic interfaces for all the JDT specific stuff.

Anything else I am missing so far?
Comment 4 Mickael Istria CLA 2019-01-29 09:39:05 EST
(In reply to Christian Pontesegger from comment #3)
> 1) I would opt for having 1 Unit Test View which could be reused by any
> language. 

or more generic a "Test" view as not all tests are unit tests ;)

> Of course we want to get rid of o.e.jdt.*

and org.junit maybe, unless we decide that the JUnit APIs directly. It could actually be a good thing to use the JUnit API for this few and let extenders notify/translate their test info as JUnit TestWatcher/Description/Result...

> I would prefer EMF compared to any hand written code here.

What would be the pros/cons here? Do you plan to generate an API with classes having the EObject as parent? Do you plan to generate mostly immutable objects with EMF (is this possible)?
It's been a while since I've generated code with EMF, so I just want to make sure it generates code which fits the requirement of extremely sustainable APIs: no noise, immutability as much as possible, simplicity, easy to edit...
  
> 4) For a first implementation I would use as much as possible from existing
> JDT code by copying relevant classes. Then we could refactor it and add
> generic interfaces for all the JDT specific stuff.

The issue with this approach is that we need to make sure and clear that everything is internal and API should not use before we are confident about the quality of the API is good enough (ie won't cost too much maintaining).
That's also a point in favor of using JUnit API which AFAIK are pretty sustainable (I could be wrong).
Comment 5 Dani Megert CLA 2019-01-29 09:48:23 EST
(In reply to Mickael Istria from comment #4)
> (In reply to Christian Pontesegger from comment #3)
> > 1) I would opt for having 1 Unit Test View which could be reused by any
> > language. 
> 
> or more generic a "Test" view as not all tests are unit tests ;)
"Test" sounds a bit too vague. How about "Test Results" view?


> > Of course we want to get rid of o.e.jdt.*
> 
> and org.junit maybe
We should definitely not depend on JUnit here. We would not have any control of future API changes/removals.


> > 4) For a first implementation I would use as much as possible from existing
> > JDT code by copying relevant classes. Then we could refactor it and add
> > generic interfaces for all the JDT specific stuff.
+1
Comment 6 Dani Megert CLA 2019-01-29 09:54:01 EST
(In reply to Christian Pontesegger from comment #3)
> 1) I would opt for having 1 Unit Test View which could be reused by any
> language. 
The big design decision depends on what we mean by "reuse". As I wrote on the mailing list:
"
The approach depends on the goal. Do we want a simple generic view that can be easily used/enhanced by a new test framework. or do want to allow each (new) test framework to provide its own look and feel inside a single view. The former will then most likely not be able to lift all feature of a given test framework, like e.g. the JUnit view currently does for JUnit.
"


> My gut feeling would say o.e.debug, but also o.e.platform.ui would
> be possible.
Debug is the right layer because it contains all the stuff for launching, and it has/allows the dependency to core.resources.
Comment 7 Christian Pontesegger CLA 2019-01-29 10:05:01 EST
(In reply to Mickael Istria from comment #4)
> > I would prefer EMF compared to any hand written code here.
> 
> What would be the pros/cons here?

Having a model in EMF allows dedicated projects like JDT to easily extend it with additional functionality. It also can be persisted nicely and the persistance layer could take care of formatting (eg. write JUnit.xml files for jenkins).
Further the model can send notifications on changes. Would allow for a nice way to inform the viewer to update.
Basically JDT would create a model root, inform the viewer about it and then JDT would simply update the model. Probably the viewer could also listen for model creation events (that's something I have to investigate in). Then viewer and model would be completely decoupled.

> > 4) For a first implementation I would use as much as possible from existing
> > JDT code
> 
> The issue with this approach is that we need to make sure and clear that
> everything is internal and API should not use before we are confident about
> the quality of the API

Currently almost everything from JDT JUnit support is internal. So this will not be a big penalty in the beginning. If we start with such a view from scratch I fear that this will never lead to a final product, so I want to keep any necessary transition for JDT as smooth as possible.
Comment 8 Christian Pontesegger CLA 2019-01-29 10:11:50 EST
(In reply to Dani Megert from comment #6)
> > 1) I would opt for having 1 Unit Test View which could be reused by any
> > language. 
> Do we want a simple generic view that can
> be easily used/enhanced by a new test framework. or do want to allow each
> (new) test framework to provide its own look and feel inside a single view.

I would prefer having 1 generic view for all kinds of tests. It would be easier for users and easier to maintain. I agree, it might be tricky to implement all kinds of specific test framework behavior.
Still I would rather drop some special functionality than having 4 differen xUnit views where all of them behave slightly different.

Anyway we should clarify these topics before I really get started. So I hope for more opinions or +1/-1 votes here.
Comment 9 Dani Megert CLA 2019-01-29 10:16:45 EST
(In reply to Christian Pontesegger from comment #8)
> (In reply to Dani Megert from comment #6)
> > > 1) I would opt for having 1 Unit Test View which could be reused by any
> > > language. 
> > Do we want a simple generic view that can
> > be easily used/enhanced by a new test framework. or do want to allow each
> > (new) test framework to provide its own look and feel inside a single view.
> 
> I would prefer having 1 generic view for all kinds of tests.
+1. Otherwise there won't be much less work to support new test frameworks and in the other approach the "only" benefit would be to have one view instead of n views. And there will still be the JUnit and other views for people who want more specific features supported.
Comment 10 Alexander Kurtakov CLA 2019-01-29 10:31:22 EST
(In reply to Dani Megert from comment #9)
> (In reply to Christian Pontesegger from comment #8)
> > (In reply to Dani Megert from comment #6)
> > > > 1) I would opt for having 1 Unit Test View which could be reused by any
> > > > language. 
> > > Do we want a simple generic view that can
> > > be easily used/enhanced by a new test framework. or do want to allow each
> > > (new) test framework to provide its own look and feel inside a single view.
> > 
> > I would prefer having 1 generic view for all kinds of tests.
> +1. Otherwise there won't be much less work to support new test frameworks
> and in the other approach the "only" benefit would be to have one view
> instead of n views. And there will still be the JUnit and other views for
> people who want more specific features supported.

Only one view it should be. One can take a look at DLTK test view (https://git.eclipse.org/c/dltk/org.eclipse.dltk.core.git/tree/core/plugins/org.eclipse.dltk.testing) for inspiration (it's overengineered IMHO but proved to work fine for Ruby (https://git.eclipse.org/c/dltk/org.eclipse.dltk.ruby.git/tree/plugins/org.eclipse.dltk.ruby.testing - multiple test frameworks hooked in) and tcl (https://git.eclipse.org/c/dltk/org.eclipse.dltk.tcl.git/tree/tcl/plugins/org.eclipse.dltk.tcl.testing https://git.eclipse.org/c/dltk/org.eclipse.dltk.tcl.git/tree/tcl/plugins/org.eclipse.dltk.tcl.testing.tcltest). So there are definitely lessons to be learned there.
Comment 11 Alexander Kurtakov CLA 2019-01-29 10:34:15 EST
As an example - here is what I had to do to add minitest support in DLTK Ruby https://git.eclipse.org/c/dltk/org.eclipse.dltk.ruby.git/log/?qt=grep&q=minitest. Most of it was my faults (learning Ruby, minitest and dltk testing support at the same time) but overall I was quite satisfied with the results.
Comment 12 Dani Megert CLA 2019-01-29 12:00:08 EST
(In reply to Alexander Kurtakov from comment #10)
> Only one view it should be.
Sure, we're talking about one view that we will create. It will be left to those who already have a related view what they do with their view. That's not the scope of this bug report.
Comment 13 Christian Pontesegger CLA 2019-03-11 10:12:14 EDT
Created attachment 277819 [details]
First proposal of a unified TestView

I have a first proposal for a generalized TestView ready. Therefore I basically took the JUnit view and ripped it out of JDT. The current proposal works buts needs quite some polishing before it is ready to be submitted to a repo.

Reason I post it is to get some feedback if the approach I chose is worthwile to continue with.

Steps to try this out:
1) Get an Oomph setup for JDT
2) import the 2 projects from this zip
3) In your run config set Auto-start =true for o.e.debug.unittest.gluecode
4) Run a unit test. Check out the new Test view


Implementation Details:
The current implementation in o.e.ui.testview uses a generic model to represent tests. Therefore some code is needed to translate JUnit test results to the new model. This code is currently part of the ...gluecode plugin and should go to JDT on the long run.


Implementation status.
The provided code is a rough version that needs polishing before it can be contributed, eg I18N, handler enablements, mnemonics, UI Resource Management, code cleanup,...
So please treat it as work in progress.

Questions:
Now for some questions.

1) Do you have any large unittest suites that do cover all  JUnit4/5 specific test features? I tried several of them, but do not have exceedingly large testsuites to try out.

2) Some features like de-/serializaton from/to xml are JUnit specific. So they need to go to JDT. To restore we would at least need a testType or similar to distinguish between JUnit/PHP/Python or any other test framework. This would be an incompatible change

3) I did not reimplement animations for the ViewPart icon/title. Not sure if this feature is really needed.

4) Toggle ignored/failed tests are available as toolbar buttons and as view menu entries. I would propose to get rid of the menu items. Its just 1 click more than the menu

5) I provided a fresh view icon and name to remove the binding to JUnit. ine with it?

6) If accepted, would a plugin org.eclipse.ui.testview be the right place and namespace to use? This will cause lots of refactoring for command names etc. therefore I would like to do this only once.

7) Most importantly: anything crucial missing? If not I would start the polishing and come up with a contribution for org.eclipse.debug
Comment 14 Mickael Istria CLA 2019-03-11 10:29:51 EDT
Thanks Christian, I didn't review it in details, but this seems on the right track.
Do you think you'd be able to also add the "glue" example for another test runner? I imagine CDT also has a view to report test, it'd be great to be able to see also how it can populate the generic view.

> 2) Some features like de-/serializaton from/to xml are JUnit specific. So
> they need to go to JDT. To restore we would at least need a testType or
> similar to distinguish between JUnit/PHP/Python or any other test framework.
> This would be an incompatible change

You're right, we need something inside the test model to identify the test framework that's used or targeted to pick the right handler or strategy.
So I believe the generic view needs to define a "testFrameworkSupport" extension point where JUnit would for example register its strategy to import/export, a filter specifying files it can import/export and similar things. Then we could maybe achieve some generic handler that according to user action and context would be able to pick the right strategy to use.
 
> 3) I did not reimplement animations for the ViewPart icon/title. Not sure if
> this feature is really needed.

I think they're useful in general, we'll eventually want them, but it's probably a bit early to emphasize on that.

> 4) Toggle ignored/failed tests are available as toolbar buttons and as view
> menu entries. I would propose to get rid of the menu items. Its just 1 click
> more than the menu

Menu items are usually pretty accessible (by keyboard or voice control). I don't know whether toolbar buttons are as good. If not, we should definitely keep both.

> 5) I provided a fresh view icon and name to remove the binding to JUnit. ine
> with it?

Didn't check the icon, but indeed, getting rid of JUnit one is a good thing.

> 6) If accepted, would a plugin org.eclipse.ui.testview be the right place
> and namespace to use? This will cause lots of refactoring for command names
> etc. therefore I would like to do this only once.

Sounds good to me, but please leave opportunity for other ones to comment too here.
Comment 15 Eclipse Genie CLA 2019-03-25 14:43:28 EDT
New Gerrit change created: https://git.eclipse.org/r/139448
Comment 16 Christian Pontesegger CLA 2019-03-25 14:44:52 EDT
To test the initial commit with JDT you will need the JDT counterpart project, which is currently availabe here:
https://github.com/Pontesegger/eclipse-testview/tree/master/org.eclipse.ui.testview.jdt
Comment 17 Christian Pontesegger CLA 2019-04-02 03:12:14 EDT
This patchset is awaiting review ...
Comment 18 Mickael Istria CLA 2019-04-02 03:28:03 EDT
I see the JUnit example seems to use an EMF model declaration to provide the concrete JUnit implementation for that view.
Is this the only way to implement a more specific model? IMO, we shouldn't require extenders to have to deal with EMF for this case.
Comment 19 Christian Pontesegger CLA 2019-04-02 07:16:34 EDT
(In reply to Mickael Istria from comment #18)
> Is this the only way to implement a more specific model? IMO, we shouldn't
> require extenders to have to deal with EMF for this case.

No, its not the only way. I am using an EMF model to store the view model. JUnit needs references to the model from JUnit to lazily retrieve stuff like stacktraces. These references can be stored in my original model. I therefore provide a generic "Object elementUnderTest".

Still it is nicer to extend the EMF model and have getters for dedicated classes.
For lazy loading we either need to extend the model or work with factories, which makes the process even more complex.
Extending the basic EMF model however could be done by simple class inheritance if implementers do not like to get familiar with EMF.

I am hoping for more discussions like this to fine tune the implementation.
Comment 20 Dani Megert CLA 2019-04-02 08:40:09 EDT
(In reply to Christian Pontesegger from comment #16)
> To test the initial commit with JDT you will need the JDT counterpart
> project, which is currently availabe here:
> https://github.com/Pontesegger/eclipse-testview/tree/master/org.eclipse.ui.testview.jdt
> 
Please provide this as Gerrit change against JDT UI (for now). I don't expect that we need a separate bundle to extend the test view.
Comment 21 Dani Megert CLA 2019-04-02 08:43:55 EDT
> If accepted, would a plugin org.eclipse.ui.testview be the right place and 
> namespace to use?
-1 for testview. As in the summary it should be testreport or testresults and follow the scheme we already use like e.g. org.eclipse.ui.views.log
==> org.eclipse.ui.views.testreport
Comment 22 Christian Pontesegger CLA 2019-04-02 09:06:34 EDT

(In reply to Dani Megert from comment #20)
> Please provide this as Gerrit change against JDT UI (for now). I don't
> expect that we need a separate bundle to extend the test view.

How can I do that? It would depend on something that does not exist yet. I guess we first need to accept the commit here, then we can push the JDT add-on to wherever JDT folks want to have it.

I also do not want to maintain the JDT integration. I provided it as proof of concept, but I guess JDT committers will want to clean that stuff up. So the github project can be seen as a first draft.

(In reply to Dani Megert from comment #21)
> > If accepted, would a plugin org.eclipse.ui.testview be the right place and 
> > namespace to use?
> -1 for testview. As in the summary it should be testreport or testresults
> and follow the scheme we already use like e.g. org.eclipse.ui.views.log
> ==> org.eclipse.ui.views.testreport

Fine with me. Bad luck I did not get that information beforehand. Refactoring will take some effort now.

Is this the only concern with the commit. If not I will wait first for other reviews.
Comment 23 Christian Pontesegger CLA 2019-04-02 09:08:00 EDT
(In reply to Dani Megert from comment #21)
> ==> org.eclipse.ui.views.testreport

BTW: its not a report (as it allows relaunch). what about
org.eclipse.ui.views.testing
Comment 24 Dani Megert CLA 2019-04-02 09:24:20 EDT
(In reply to Christian Pontesegger from comment #22)
> 
> (In reply to Dani Megert from comment #20)
> > Please provide this as Gerrit change against JDT UI (for now). I don't
> > expect that we need a separate bundle to extend the test view.
> 
> How can I do that? It would depend on something that does not exist yet.
It will exist in your workspace and depend on the other change. Of course it will not get a +1 from Jenkins.


> I guess we first need to accept the commit here, then we can push the JDT
> add-on to wherever JDT folks want to have it.
-2. No way to merge something that does not have a working and accepted proof of concept, e.g. in JDT (or some other tool).

 
> I also do not want to maintain the JDT integration. I provided it as proof
> of concept, but I guess JDT committers will want to clean that stuff up.
At least from my team there is no one. We have the working JUnit view. To it looks more reasonable to pick a tool that does not yet have a Test Results view.


> (In reply to Dani Megert from comment #21)
> > > If accepted, would a plugin org.eclipse.ui.testview be the right place and 
> > > namespace to use?
> > -1 for testview. As in the summary it should be testreport or testresults
> > and follow the scheme we already use like e.g. org.eclipse.ui.views.log
> > ==> org.eclipse.ui.views.testreport
> 
> Fine with me. Bad luck I did not get that information beforehand.
> Refactoring will take some effort now.
> 
> Is this the only concern with the commit. If not I will wait first for other
> reviews.
I did not look at the change, so, for sure can't claim that this is the only concern.
Comment 25 Dani Megert CLA 2019-04-02 09:25:26 EDT
(In reply to Christian Pontesegger from comment #23)
> (In reply to Dani Megert from comment #21)
> > ==> org.eclipse.ui.views.testreport
> 
> BTW: its not a report (as it allows relaunch). what about
> org.eclipse.ui.views.testing
I don't like to use the name "test" or "testing". What's the problem with relaunch? Relaunch will just generate a new report. Test Results (view) would also work for me.
Comment 26 Christian Pontesegger CLA 2019-04-02 10:40:14 EDT
(In reply to Dani Megert from comment #24)
> It will exist in your workspace and depend on the other change. Of course it
> will not get a +1 from Jenkins.

ok, will do.

> -2. No way to merge something that does not have a working and accepted
> proof of concept, e.g. in JDT (or some other tool).

I already provided the proof of concept. Do we need JDT to FIRST accept the proof of concept before we can merge this one? Seems like a chicken and egg problem.

> At least from my team there is no one. We have the working JUnit view. To it
> looks more reasonable to pick a tool that does not yet have a Test Results
> view.

If JDT is not willing to move to the new view (and that should be done by a JDT maintainer) then this whole topic will not succeed.
The whole bug is about uniting 5+ different Unit Test views into a generic one. JDT would be the lighthouse project.

I am offering help for JDT adopters, but JUnit is evolving and so will the view. Therefore JDT maintainers should know their ways around in the code.

> I did not look at the change, so, for sure can't claim that this is the only
> concern.

Who would be willing to review the change?
Comment 27 Dani Megert CLA 2019-04-02 10:47:17 EDT
(In reply to Christian Pontesegger from comment #26)
> (In reply to Dani Megert from comment #24)
> > It will exist in your workspace and depend on the other change. Of course it
> > will not get a +1 from Jenkins.
> 
> ok, will do.
> 
> > -2. No way to merge something that does not have a working and accepted
> > proof of concept, e.g. in JDT (or some other tool).
> 
> I already provided the proof of concept. Do we need JDT to FIRST accept the
> proof of concept before we can merge this one? Seems like a chicken and egg
> problem.
Yes, like with adding APIs, there should be a client that tested the API and confirms it is working. So far, no client (not counting you as client) has confirmed that the framework is OK. What you need, is someone who is eager to use that stuff, work with you, and then both things can be merged together. Otherwise we end up with the new view being merged but no one is using it. Again, I'm sorry to say, but from my JDT team no one has time at the moment for this. If JDT is your client, you need to find some other JDT committer that works with you.


> > At least from my team there is no one. We have the working JUnit view. To it
> > looks more reasonable to pick a tool that does not yet have a Test Results
> > view.
> 
> If JDT is not willing to move to the new view (and that should be done by a
> JDT maintainer) then this whole topic will not succeed.
> The whole bug is about uniting 5+ different Unit Test views into a generic
> one. JDT would be the lighthouse project.
I did not say JDT is not willing. I said that from my team there's no capacity. If you can find another JDT committer, that's fine.


> I am offering help for JDT adopters, but JUnit is evolving and so will the
> view. Therefore JDT maintainers should know their ways around in the code.
> 
> > I did not look at the change, so, for sure can't claim that this is the only
> > concern.
> 
> Who would be willing to review the change?
There are already many on cc, but I suggest you ask on jdt-dev. The first step would be to try it out before doing any review.
Comment 28 Dani Megert CLA 2019-04-02 11:17:23 EDT
Just a clarification: while JDT can contribute and use the new view JDT will not remove the JUnit view. So, the contribution to JDT must not replace or hide the JUnit view, or alter it in any way.
Comment 29 Christian Pontesegger CLA 2019-04-02 16:58:52 EDT
Ok guys, you lost me on the strategy here.

I thought this is all about:
* having a generic view to display tests
* having a joint and more fluent user experience for anybody using more than 1 language in eclipse
* having less maintenance effort as 1 view should be less work than 5
* simplify life for new languages as they can reuse existing stuff

I put effort into the commit to mimic all the behavior from the JUnit view. Now both views look identical and behave identical (only thing missing is command enablement/disablement depending on the testsuite run state)

Now I learn that JDT will not replace the JUnit view, even if the new view looks identical. Why on earth would any user use a generic view if there is a specialized one? Why would JDT team put any improvements in the generic view when they have their own specialized one?

And most interesting: why should I find someone on JDT to rate the view implementation if they will anyway not adopt?

To me this does not make sense at all. Furthermore without anybody willing to review and finally push the commit I do not know how to continue. So lets wait if somebody steps up.
Comment 30 Christian Pontesegger CLA 2019-04-02 17:00:01 EDT
Created attachment 278126 [details]
Old/new view comparison

Just in case you want to know what it looks like...
Comment 31 Mickael Istria CLA 2019-04-03 03:59:35 EDT
Christian, I think your frustration comes from the fact that you've set as main goal to replace JUnit view soon. As any change in JDT legacy, it's not so simple. JDT has legacies and things such like menus or toolbars or other stuff wouldn't be so easy to adopt in the new view, nor to port to the new view.
I think you should keep focus and motivation on the initial goal of this ticket: providing a generic test report view. When this emerges and is good enough, I'm pretty confident that several projects, other than JDT, will be happy to use this factorized piece.

About the review, I'm still not sure I understand how can one implement some new Test framework. While I see some interesting classes and a lot of handlers, I don't see a kind of "service" that would allow to easily write the interaction between the view and the test framework.
I would typically look for something like a "TestViewerService" API, with some methods like "startNewSession()", rerunTests(test...), notifyTestStarted(Test), notifyTestFailed(...) and so on which I could use in my test framework invocation to have the view reflecting changes as we run it. Is such service existing already?
Also, in the test view code, everything seems to be exported as API. That seems unsafe: the more classes are exposed as APIs, the more expensive the maintenance will be. So it would be nice if you restrict the API to the minimal stuff to make things work. I find at the moment that the specialization like the one you demonstrate for JUnit still requires a lot of code. It would be great if it could be substantially less, so people who need to adopt a test view will clearly find the new generic view more appealing than copy-pasting JUnit view and tweaking it to their needs. Moreover, I'd like to make sure that extending the EMF model is *not* mandatory nor necessary in the general case. Those layers of code generation introduce some complexity in the project, some complexity that many plugin developers will try to avoid as much as possible.
Comment 32 Dani Megert CLA 2019-04-03 04:12:38 EDT
It might have sounded different, but let me say that I like the idea to have this Test Report/Results view.

I also think using the JUnit as inspiration is a good idea, like TestNG was inspired by JUnit. However, that does not mean that replacing the existing JUnit view itself needs to be part of this. You could offer the extension for JUnit e.g. via Marketplace.

I like the idea that Mickael posted on the mailing list: "Maybe you can try adding support for something like TestNG instead of JUnit". That would add value to JDT.
Comment 33 Dani Megert CLA 2019-04-03 04:13:45 EDT
(In reply to Christian Pontesegger from comment #30)
> Created attachment 278126 [details]
> Old/new view comparison
> 
> Just in case you want to know what it looks like...
Looks nice! I would like to see in the view with which test framework (e.g. JUnit) the tests were run.
Comment 34 Christian Pontesegger CLA 2019-04-03 15:45:17 EDT
(In reply to Mickael Istria from comment #31)

> About the review, I'm still not sure I understand how can one implement some
> new Test framework. While I see some interesting classes and a lot of
> handlers, I don't see a kind of "service" that would allow to easily write
> the interaction between the view and the test framework.

Currently I did not offer such a service directly. Basically it all works by using the EMF model classes. Lets have a look at the JUnit integration:

It starts with the activator:
https://github.com/Pontesegger/eclipse-testview/blob/master/org.eclipse.ui.testview.jdt/src/org/eclipse/jdt/JUnitGluePlugin.java
line 82: here we register a listener to the JUnit framework. The listener will then create EMF model classes, eg here:

https://github.com/Pontesegger/eclipse-testview/blob/master/org.eclipse.ui.testview.jdt/src/org/eclipse/jdt/TestRunSessionListener.java
line 154: create a new ITestRoot instance.You might use the EMF model from the generic view directly. For JUnit I derived from the classes to nicely store references to JUnit specific classes.

https://git.eclipse.org/r/c/139448/2/org.eclipse.ui.testview/src/org/eclipse/ui/testview/TestViewPlugin.java line 208: takes over the role of the service you are asking. I agree, it would be nicer to carve this out and use a service for this.

Once the notification is sent, the unit view will pick it up and display its content. Changes to the model will be recognized by the view via EMF notifications automatically. Only thing left for the JUnit integration is to take care that the view gets opened in case unit tests do get started. But this is done for the current JUnit view, too.

> Also, in the test view code, everything seems to be exported as API. That
> seems unsafe: the more classes are exposed as APIs, the more expensive the
> maintenance will be. So it would be nice if you restrict the API to the
> minimal stuff to make things work.

Only classes needed are the model interfaces, the factory and 2-3 other interfaces in a separate package. We can hide the rest, no problem.

> I find at the moment that the
> specialization like the one you demonstrate for JUnit still requires a lot
> of code. It would be great if it could be substantially less, so people who
> need to adopt a test view will clearly find the new generic view more
> appealing than copy-pasting JUnit view and tweaking it to their needs.

Agreed, it should be compact. The required glue code for JDT is currently only 2 files:
https://github.com/Pontesegger/eclipse-testview/blob/master/org.eclipse.ui.testview.jdt/src/org/eclipse/jdt/TestRunSessionListener.java ... to convert from your unit framework to the view model.

https://github.com/Pontesegger/eclipse-testview/blob/master/org.eclipse.ui.testview.jdt/src/org/eclipse/jdt/JUnitTestRunner.java ... for launching and relaunching stuff

The rest of the glue plugin is JUnit/JDT specific stuff like finding java classes for doubleclick actions, stacktrace parsing and many things more I did not look into. This was just a copy and paste from existing JDT code.

> Moreover, I'd like to make sure that extending the EMF model is *not*
> mandatory nor necessary in the general case. Those layers of code generation
> introduce some complexity in the project, some complexity that many plugin
> developers will try to avoid as much as possible.

Agreed. The basic model would serve just fine. In that case the model would have to be fully populated (eg stacktraces), which might be expensive. Another option would be to support lazy loading using factories or proxies, but this makes to code complicated, too.
Finally users could directly inherit from EMF classes to support lazy loading stuff, then we would have to expot EMF implementations, too.

For JDT I chose the option the extend the EMF model as it was quicker to implement this way, but its not necessary at all.
Comment 35 Andrey Loskutov CLA 2019-04-03 15:49:16 EDT
Just a side note: I haven't reviewed the code, but I'm a bit worried about EMF model being used - AFAIK EMF is not MT safe, but most of debug tools are usually heavy on parallel tasks etc.
Comment 36 Christian Pontesegger CLA 2019-04-03 15:55:02 EDT
(In reply to Dani Megert from comment #32)
> You could offer the extension
> for JUnit e.g. via Marketplace.

Does not make sense. First of all I do not want to offer a new product. The intention was to reunite at least some of the Unit Test View forks out there, not to create another one, as stated on the mailing list.

> I like the idea that Mickael posted on the mailing list: "Maybe you can try
> adding support for something like TestNG instead of JUnit". That would add
> value to JDT.

Yes, I agree. But that part I would leave to someone else.

The testview is something I need for my daily work on EASE. I have my own fork of a Unit view there too and was hoping to get rid of it by having something more generic.
Comment 37 Eclipse Genie CLA 2020-10-22 08:33:08 EDT
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.debug/+/171116
Comment 38 Eclipse Genie CLA 2020-10-22 10:24:36 EDT
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.ui/+/171126
Comment 39 Victor Rubezhny CLA 2020-10-22 10:31:35 EDT
(In reply to Eclipse Genie from comment #37)
> New Gerrit change created:
> https://git.eclipse.org/r/c/platform/eclipse.platform.debug/+/171116

This change proposes org.eclipse.unittest bundle that contains an alternative Test Result View (also based on the JUnit one) that is about as feature-rich as the JUnit one. 

There are few Unit Test bundle clients ready (ready at least to demonstrate the view's functionality) for JUnit, C/C++ and Corrosion/Rust (Cargo) unit test frameworks.

We believe that the current proposal, and more specifically the APIs,  is now of good enough quality to be merged in Platform.

Yet another Gerrit change contains org.eclipse.unittest.junit bundle - a JUnit test runner client that demonstrates the Unit Test bundle functionality: https://git.eclipse.org/r/c/jdt/eclipse.jdt.ui/+/171126
Comment 40 Eclipse Genie CLA 2020-10-22 13:27:00 EDT
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.debug/+/171138
Comment 42 Eclipse Genie CLA 2020-11-10 07:31:55 EST
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.releng.aggregator/+/172037
Comment 44 Sarika Sinha CLA 2020-11-17 23:12:36 EST
We want to move this bug to RC1 or close this one and create new for RC1 ?
Comment 45 Mickael Istria CLA 2020-11-18 03:05:06 EST
(In reply to Sarika Sinha from comment #44)
> We want to move this bug to RC1 or close this one and create new for RC1 ?

It depends: do you think it would be fine to add the JDT bundle that integrates in the view by RC1? This bundle wouldn't be part of default SDK nor other products (only in the p2 repo for testing) and doesn't expose any API.
So if it's OK to have it merged in between M3 and RC1, we should move this bug to RC1; if it's preferred to delay to 4.19, we can open a new bug and close this one. Your call ;)
Comment 46 Mickael Istria CLA 2020-11-18 03:27:36 EST
We'll also need to update the documentation and add a N&N entry to fully cover this bug. I can try to work on and submit Gerrit patches today.
Comment 47 Lars Vogel CLA 2020-11-18 03:45:25 EST
Is it planned to replace JDT JUnit test view with the new one?
Comment 48 Alexander Kurtakov CLA 2020-11-18 04:28:53 EST
(In reply to Lars Vogel from comment #47)
> Is it planned to replace JDT JUnit test view with the new one?

Not really. The goal is to keep enhancing it with missing features that JUnit view provides. If/when feature parity is reached such a discussion can be held but right now this is not a goal.
Comment 49 Sarika Sinha CLA 2020-11-18 06:53:15 EST
(In reply to Mickael Istria from comment #45)
> (In reply to Sarika Sinha from comment #44)
> > We want to move this bug to RC1 or close this one and create new for RC1 ?
> 
> It depends: do you think it would be fine to add the JDT bundle that
> integrates in the view by RC1? This bundle wouldn't be part of default SDK
> nor other products (only in the p2 repo for testing) and doesn't expose any
> API.
> So if it's OK to have it merged in between M3 and RC1, we should move this
> bug to RC1; if it's preferred to delay to 4.19, we can open a new bug and
> close this one. Your call ;)

If it is only in the p2 repo, we can add in RC1 also. 
Moving the bug to RC1.
Comment 50 Eclipse Genie CLA 2020-11-18 12:04:02 EST
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.common/+/172450
Comment 51 Eclipse Genie CLA 2020-11-18 12:59:44 EST
New Gerrit change created: https://git.eclipse.org/r/c/www.eclipse.org/eclipse/news/+/172466
Comment 55 Jeff Johnston CLA 2020-11-20 16:00:39 EST
Released for 4.18RC1
Comment 56 Eclipse Genie CLA 2020-11-21 01:34:33 EST
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.debug/+/172605
Comment 58 Eclipse Genie CLA 2020-11-21 15:46:22 EST
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.common/+/172630
Comment 60 Andrey Loskutov CLA 2020-11-23 04:19:43 EST
UA tests are failing in https://download.eclipse.org/eclipse/downloads/drops4/I20201122-1800/testresults/html/org.eclipse.ua.tests.doc_ep418I-unit-cen64-gtk3-java11_linux.gtk.x86_64_11.html

estTopicsReference	Failure	expected:<[]> but was:<[ * Unexpected extension points in topics_Reference.xml: org.eclipse.unittest.ui.unittestViewSupport * Unexpected exported API packages in topics_Reference.xml: org.eclipse.unittest.launcher org.eclipse.unittest.model org.eclipse.unittest.ui ]>

org.junit.ComparisonFailure: expected:<[]> but was:<[
* Unexpected extension points in topics_Reference.xml:
org.eclipse.unittest.ui.unittestViewSupport

* Unexpected exported API packages in topics_Reference.xml:
org.eclipse.unittest.launcher
org.eclipse.unittest.model
org.eclipse.unittest.ui
]>

Please check that.
Comment 61 Mickael Istria CLA 2020-11-23 04:38:29 EST
(In reply to Andrey Loskutov from comment #60)
> UA tests are failing in

Thanks Andrey, I was actually working on it just now!
Comment 62 Eclipse Genie CLA 2020-11-24 04:01:06 EST
New Gerrit change created: https://git.eclipse.org/r/c/platform/eclipse.platform.releng.aggregator/+/172733
Comment 64 Eclipse Genie CLA 2020-11-24 13:51:06 EST
New Gerrit change created: https://git.eclipse.org/r/c/cdt/org.eclipse.cdt/+/172779
Comment 65 Sarika Sinha CLA 2020-11-25 00:23:15 EST
(In reply to Eclipse Genie from comment #64)
> New Gerrit change created:
> https://git.eclipse.org/r/c/cdt/org.eclipse.cdt/+/172779

I guess we are done with this bug.
We can take up the remaining things in a followup bug.
Comment 66 Sarika Sinha CLA 2020-11-26 06:17:45 EST
(In reply to Eclipse Genie from comment #52)
> Gerrit change
> https://git.eclipse.org/r/c/www.eclipse.org/eclipse/news/+/172466 was merged
> to [master].
> Commit:
> http://git.eclipse.org/c/www.eclipse.org/eclipse/news.git/commit/
> ?id=572d3bde186816a9579ec1113c8724c0263d954b

@Michael, 
N&N does not mention that it is not available with SDK and only with p2, should we add that?
Comment 67 Mickael Istria CLA 2020-11-30 22:44:46 EST
(In reply to Sarika Sinha from comment #66)
> @Michael, 
> N&N does not mention that it is not available with SDK and only with p2,
> should we add that?

I see
"""
The org.eclipse.unittest.ui plugin is currently not included in Eclipse Platform, SDK or other products; it's available for installation from the Eclipse project p2 repository. Consumers would typically need to include this bundle in there target-platform explicitly to provide an extension for the Unit Test view.
"""
Anything more specific you think needs to be added?
Comment 68 Sarika Sinha CLA 2020-11-30 23:43:27 EST
(In reply to Mickael Istria from comment #67)
> (In reply to Sarika Sinha from comment #66)
> > @Michael, 
> > N&N does not mention that it is not available with SDK and only with p2,
> > should we add that?
> 
> I see
> """
> The org.eclipse.unittest.ui plugin is currently not included in Eclipse
> Platform, SDK or other products; it's available for installation from the
> Eclipse project p2 repository. Consumers would typically need to include
> this bundle in there target-platform explicitly to provide an extension for
> the Unit Test view.
> """
> Anything more specific you think needs to be added?

Looks good. Thanks.
Comment 69 Mickael Istria CLA 2020-12-08 05:24:00 EST
As a followup, here is a video demo of how it looks like with Corrosion: ▶️ https://www.screencast.com/t/1sgBo0ENGc
Comment 70 Lars Vogel CLA 2020-12-08 05:40:38 EST
(In reply to Mickael Istria from comment #69)
> As a followup, here is a video demo of how it looks like with Corrosion: ▶️
> https://www.screencast.com/t/1sgBo0ENGc

Looks great. Thanks!
Comment 71 Matthias Becker CLA 2020-12-22 09:11:30 EST
platform.ua gerrrit build currently fails. For an example see:

https://ci.eclipse.org/platform/job/eclipse.platform.ua-Gerrit/1471/console

This seems related to the new org.eclipse.unittest.ui plugin.


The log contains:

testTopicsReference(org.eclipse.ua.tests.doc.internal.linkchecker.ApiDocTest)  Time elapsed: 0.329 s  <<< FAILURE!
org.junit.ComparisonFailure: 
expected:<[]> but was:<[
* Unexpected extension points in topics_Reference.xml:
org.eclipse.unittest.ui.unittestViewSupport

* Unexpected exported API packages in topics_Reference.xml:
org.eclipse.unittest.launcher
org.eclipse.unittest.model
org.eclipse.unittest.ui
]>
	at org.eclipse.ua.tests.doc.internal.linkchecker.ApiDocTest.testTopicsReference(ApiDocTest.java:183)

testAllLinks(org.eclipse.ua.tests.doc.internal.linkchecker.LinkTest)  Time elapsed: 2.687 s

Results:

Failures: 
  ApiDocTest.testTopicsReference:183 expected:<[]> but was:<[
* Unexpected extension points in topics_Reference.xml:
org.eclipse.unittest.ui.unittestViewSupport

* Unexpected exported API packages in topics_Reference.xml:
org.eclipse.unittest.launcher
org.eclipse.unittest.model
org.eclipse.unittest.ui
]>

I was not able to re-produce this locally on my machine. Can you pls. have a look at it?
Comment 72 Mickael Istria CLA 2021-01-04 11:06:37 EST
(In reply to Matthias Becker from comment #71)
> I was not able to re-produce this locally on my machine. Can you pls. have a
> look at it?

Was this patch rebased on top of master? This might be some out-of-date build config that miss newer plugins.