Bug 497871 - Provide a Generic and extensible Text Editor
Summary: Provide a Generic and extensible Text Editor
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Text (show other bugs)
Version: 4.6   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: 4.7 M3   Edit
Assignee: Mickael Istria CLA
QA Contact:
URL:
Whiteboard:
Keywords: noteworthy
: 181907 496115 496117 (view as bug list)
Depends on: 500755
Blocks: 499810 499811 499812
  Show dependency tree
 
Reported: 2016-07-13 16:35 EDT by Mickael Istria CLA
Modified: 2016-10-26 11:44 EDT (History)
13 users (show)

See Also:


Attachments
Ant verbose log for generic editor testrun (149.56 KB, text/plain)
2016-10-01 14:40 EDT, Sravan Kumar Lakkimsetti CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mickael Istria CLA 2016-07-13 16:35:09 EDT
Most existing Text Editors (Java, XML, JS...) had to reimplement support for extensions to support additional content assist, hover, syntax operations coming from external sources.
In the same way, Eclipse should offer a generic and extensible Text Editors, that could be enriched directly by 3rd-party extensions. That would allow to more easily bootstrap and extend an editor simply by providing extensions to it for given file formats. Then from user perspective, this would appear as a unified and consistent editors for multiple languages.
The "features" of the editor can be hooked from multiple source, such as a TextMate grammar for coloration and some language service for completion. Having extensions for each of the sub-features of an editor makes it easier to consume features from multiple sources.
This editor should be open when it has some features for some languages that aren't supported otherwise in the IDE.
Comment 1 Mickael Istria CLA 2016-07-13 17:30:14 EDT
Gerrit patch https://git.eclipse.org/r/75921 contains a proposal of such editor, with some tests and examples (an editor for .project) to show how to take advantage of those extensions.
Comment 2 Vlad Dumitrescu CLA 2016-08-13 11:11:27 EDT
This is a wonderful initiative! I suppose that the goal is that all language services will be possible to install as extension points.

* Would it be too much to ask to make the implementation as backwards-compatible as possible? That would mean 2 versions back (currently 4.4), but even just one might be ok if that isn't possible.

* Do you have some kind of time plan?
Comment 3 Vlad Dumitrescu CLA 2016-08-13 11:32:49 EDT
I'm not sure how the process works @ Eclipse, I made some comments on Gerrit, should they have been done here? If I want to supply some updated code, how does that work when the baseline is a set of Gerrit patches? This is a very early phase, so it can get confusing very fast if there are many changes depending on each other from different people...
Comment 4 Vlad Dumitrescu CLA 2016-08-13 13:08:03 EDT
What if instead of many separate extension points, there was just one with multiple leafs, roughly corresponding to a "language server"? I'm not sure if that allows third party plugins to provide only pieces of such a server.
Comment 5 Lars Vogel CLA 2016-08-15 07:55:56 EDT
(In reply to Vlad Dumitrescu from comment #3)
> I'm not sure how the process works @ Eclipse, I made some comments on
> Gerrit

I do not see comments from you in Gerrit. Did you publish them?

>  to make the implementation as backwards-compatible as possible

I believe such big new features will be only done for the new release and not downported.
Comment 6 Mickael Istria CLA 2016-08-16 04:37:31 EDT
(In reply to Vlad Dumitrescu from comment #2)
> * Would it be too much to ask to make the implementation as
> backwards-compatible as possible? That would mean 2 versions back (currently
> 4.4), but even just one might be ok if that isn't possible.

As we're implementing and designing this, there is no focus on backward compatibility at the moment, and I believe that neither Sopot nor I have plans to work on it. That said, if you find small changes that allow backward compatibility without any drawback, they'd be totally welcome. Commenting on Gerrit is the right approach to suggest such changes at the moment.
Once the initial code got merged, you'll be able to use the regular Gerrit process to propose change: https://wiki.eclipse.org/Platform_UI/How_to_Contribute#Creating_a_Gerrit_review_or_a_patch

> * Do you have some kind of time plan?

The sooner the better, but all that depends on the availability of other contributors to review this; and that cannot be easily estimated.
The first release with such editor would be Oxygen, by June 2017; but we hope we'll manage to have a first version of this editor in next milestone (Oxygen milestone 2, September).
Keep on reading notifications for this bugzilla so you'll get informed when the code is merged.

(In reply to Vlad Dumitrescu from comment #4)
> What if instead of many separate extension points, there was just one with
> multiple leafs, roughly corresponding to a "language server"? I'm not sure
> if that allows third party plugins to provide only pieces of such a server.

As those extension points are pretty generic, they might easily be reused by some other editors out there. Smaller grain extension points allow an easier reusability and are IMO a bit simpler to maintain.
Comment 7 Vlad Dumitrescu CLA 2016-08-17 10:51:00 EDT
I understand, I was coming from a different place so I'll have to get used to the new way of thinking. Thanks!
Comment 8 Eclipse Genie CLA 2016-08-24 10:07:05 EDT
New Gerrit change created: https://git.eclipse.org/r/79630
Comment 9 Dani Megert CLA 2016-09-06 08:39:13 EDT
It's going positively into the right direction but needs work. We should commit it early M3.
Comment 10 Eclipse Genie CLA 2016-09-06 16:51:29 EDT
New Gerrit change created: https://git.eclipse.org/r/80505
Comment 11 Dani Megert CLA 2016-09-08 04:44:49 EDT
The new plug-in needs to be added to the doc plug-in. For details see https://wiki.eclipse.org/How_to_add_things_to_the_Eclipse_doc

We also have to
- add the new bundles to the corresponding features
- add the bundles to the official build, so that they become part of the download
- invoke the tests during our official build

The last item will require Releng to make the changes. The other ones should be doable by you guys.
Comment 12 Lars Vogel CLA 2016-09-08 05:02:08 EDT
IMHO the items from Comment 11 should be handled via separate bugs. I think we should have very focused bugs and especially help should not be blocking.
Comment 13 Eclipse Genie CLA 2016-09-08 05:39:01 EDT
New Gerrit change created: https://git.eclipse.org/r/80674
Comment 14 Alexander Kurtakov CLA 2016-09-08 06:00:58 EDT
(In reply to Lars Vogel from comment #12)
> IMHO the items from Comment 11 should be handled via separate bugs. I think
> we should have very focused bugs and especially help should not be blocking.

I fully agree on this one as most of the additional things can't really happen before the code in question is in.
Dani or Sravan, do you have pointers to adding the tests? I'm eager to see this going and to try my new releng commit rights with this one.
Comment 15 Dani Megert CLA 2016-09-08 09:22:17 EDT
(In reply to Alexander Kurtakov from comment #14)
> (In reply to Lars Vogel from comment #12)
> > IMHO the items from Comment 11 should be handled via separate bugs. I think
> > we should have very focused bugs and especially help should not be blocking.
> 
> I fully agree on this one as most of the additional things can't really
> happen before the code in question is in.

No, but together. And the Gerrit changes can be uploaded now. There is no point to rush if the code is not available for users as part of our builds.
Comment 16 Dani Megert CLA 2016-09-08 09:22:47 EDT
(In reply to Alexander Kurtakov from comment #14)
> Dani or Sravan, do you have pointers to adding the tests? I'm eager to see
> this going and to try my new releng commit rights with this one.

Did you look in the Releng FAQs and wiki?
Comment 17 Alexander Kurtakov CLA 2016-09-08 09:29:29 EDT
(In reply to Dani Megert from comment #16)
> (In reply to Alexander Kurtakov from comment #14)
> > Dani or Sravan, do you have pointers to adding the tests? I'm eager to see
> > this going and to try my new releng commit rights with this one.
> 
> Did you look in the Releng FAQs and wiki?

Yes, what I found is https://wiki.eclipse.org/Platform-releng-faq-obsolete#How_do_I_contribute_a_JUnit_Plug-in_Test_to_the_build.3F which has "The Eclipse release engineering team will update the appropriate feature and scripts to build and run the new tests. " so I was looking for some kind of list of those.
Comment 18 Alexander Kurtakov CLA 2016-09-08 09:34:15 EDT
And of course I went to the "obsolete" page (sigh). But even https://wiki.eclipse.org/Platform-releng/Platform_Build#Running_the_Eclipse_platform_tests didn't give me an idea of what exactly is needed from releng to do.
Comment 19 Sravan Kumar Lakkimsetti CLA 2016-09-09 03:23:26 EDT
(In reply to Alexander Kurtakov from comment #18)
> And of course I went to the "obsolete" page (sigh). But even
> https://wiki.eclipse.org/Platform-releng/
> Platform_Build#Running_the_Eclipse_platform_tests didn't give me an idea of
> what exactly is needed from releng to do.

Please have a look at this file https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/production/testScripts/configuration/sdk.tests/testScripts/test.xml#n2184

This lists the tests that needs to be run during the build.
Comment 20 Mickael Istria CLA 2016-09-20 06:20:30 EDT
When can we hope to have this patch merged in platform.text repo?
About releng, which feature(s) should include this bundle? Only the regular one or the .project editor and example should be published on a p2 repo too?
Comment 21 Eclipse Genie CLA 2016-09-21 10:32:55 EDT
New Gerrit change created: https://git.eclipse.org/r/81599
Comment 22 Dani Megert CLA 2016-09-26 05:08:05 EDT
(In reply to Dani Megert from comment #11)
> The new plug-in needs to be added to the doc plug-in. For details see
> https://wiki.eclipse.org/How_to_add_things_to_the_Eclipse_doc

This is in progress via https://git.eclipse.org/r/#/c/80674/


> We also have to
> - add the new bundles to the corresponding features

This is in progress via https://git.eclipse.org/r/#/c/81599/


> - add the bundles to the official build, so that they become part of the
> download

Looks good (root pom.xml is updated).


> - invoke the tests during our official build

A change for that is still need, see comment 19.
Comment 23 Eclipse Genie CLA 2016-09-26 06:46:43 EDT
New Gerrit change created: https://git.eclipse.org/r/81891
Comment 24 Lars Vogel CLA 2016-09-27 10:34:50 EDT
Would be nice to have a PDE template for this new API. Opened Bug 502258 for that.
Comment 31 Dani Megert CLA 2016-09-28 16:36:46 EDT
Status update after merging all changes:
- build went without breaking
- code is in the downloads
- example is in the download
- tests are in the download
- doc looks good except for a typo that I fixed with http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/?id=ed21e9f2cfeffcee7cda40ecf79fe4b7592aa4ca
- PROBLEM: tests did not get executed or published during our official build

Sravan, it looks like your comment was incomplete. Maybe the tests were run but not published? Please follow up.
Comment 32 Sravan Kumar Lakkimsetti CLA 2016-09-29 02:19:06 EDT
(In reply to Dani Megert from comment #31)
> Status update after merging all changes:
> - build went without breaking
> - code is in the downloads
> - example is in the download
> - tests are in the download
> - doc looks good except for a typo that I fixed with
> http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/
> ?id=ed21e9f2cfeffcee7cda40ecf79fe4b7592aa4ca
> - PROBLEM: tests did not get executed or published during our official build
> 
> Sravan, it looks like your comment was incomplete. Maybe the tests were run
> but not published? Please follow up.

The tests did not ran. here is the error message I got
Warning: Could not find file /Users/hudsonBuild/workspace/ep47N-unit-mac64/workarea/N20160928-0710/eclipse-testing/test-eclipse/Eclipse.app/Contents/Eclipse/org.eclipse.ui.genericeditor.tests.xml to copy

I will investigate furthur
Comment 33 Nobody - feel free to take it CLA 2016-09-29 03:08:55 EDT
> The tests did not ran. here is the error message I got
> Warning: Could not find file
> /Users/hudsonBuild/workspace/ep47N-unit-mac64/workarea/N20160928-0710/
> eclipse-testing/test-eclipse/Eclipse.app/Contents/Eclipse/org.eclipse.ui.
> genericeditor.tests.xml to copy
> 
> I will investigate furthur

Shouldn't the URL end with something similar to .../org.eclipse.ui.genericeditor.tests/test.xml ?
Comment 34 Sravan Kumar Lakkimsetti CLA 2016-09-29 03:52:13 EDT
(In reply to Sopot Cela from comment #33)
> > The tests did not ran. here is the error message I got
> > Warning: Could not find file
> > /Users/hudsonBuild/workspace/ep47N-unit-mac64/workarea/N20160928-0710/
> > eclipse-testing/test-eclipse/Eclipse.app/Contents/Eclipse/org.eclipse.ui.
> > genericeditor.tests.xml to copy
> > 
> > I will investigate furthur
> 
> Shouldn't the URL end with something similar to
> .../org.eclipse.ui.genericeditor.tests/test.xml ?

I think you are right. I will see why tests are looking for org.eclipse.ui.genericeditor.tests.xml
Comment 35 Sravan Kumar Lakkimsetti CLA 2016-09-29 07:51:41 EDT
In .../org.eclipse.ui.genericeditor.tests/test.xml the plugin-name was given as org.eclipse.ui.editors.tests and in 
aggregator repo the pluginname is defined as org.eclipse.ui.genericeditor.tests

Can you please correct this?
Comment 36 Eclipse Genie CLA 2016-09-29 07:54:44 EDT
New Gerrit change created: https://git.eclipse.org/r/82156
Comment 37 Mickael Istria CLA 2016-09-29 07:56:27 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #35)
> In .../org.eclipse.ui.genericeditor.tests/test.xml the plugin-name was given
> as org.eclipse.ui.editors.tests and in 
> aggregator repo the pluginname is defined as
> org.eclipse.ui.genericeditor.tests

Thanks for the inverstigation and guidance.
I've submitted https://git.eclipse.org/r/82156 that should fix it. Would you mind to review it?
Comment 39 Mickael Istria CLA 2016-09-29 11:54:18 EDT
*** Bug 496117 has been marked as a duplicate of this bug. ***
Comment 40 Mickael Istria CLA 2016-09-29 11:58:57 EDT
*** Bug 496115 has been marked as a duplicate of this bug. ***
Comment 41 Sravan Kumar Lakkimsetti CLA 2016-09-30 04:48:26 EDT
The tests did not run in N20160930-0230 also. I am investigating it
Comment 42 Nobody - feel free to take it CLA 2016-09-30 04:53:03 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #41)
> The tests did not run in N20160930-0230 also. I am investigating it

What was the error this time?
Comment 43 Nobody - feel free to take it CLA 2016-09-30 04:59:48 EDT
(In reply to Sopot Cela from comment #42)
> (In reply to Sravan Kumar Lakkimsetti from comment #41)
> > The tests did not run in N20160930-0230 also. I am investigating it
> 
> What was the error this time?

I think I know:

https://git.eclipse.org/c/platform/eclipse.platform.text.git/tree/org.eclipse.ui.genericeditor.tests/test.xml?id=4de07cf7328f19fab490c4c93b34aca5ef00c30b#n32 should have been  org.eclipse.ui.genericeditor.tests.GenericEditorTestSuite

Gerrit incoming
Comment 44 Eclipse Genie CLA 2016-09-30 05:05:03 EDT
New Gerrit change created: https://git.eclipse.org/r/82245
Comment 47 Sravan Kumar Lakkimsetti CLA 2016-09-30 13:54:04 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #46)
> Here is the error message I am getting
> https://hudson.eclipse.org/shared/job/ep47N-unit-mac64/ws/workarea/N20160930-
> 0900/eclipse-testing/results/html/org.eclipse.releng.tests_ep47N-unit-
> mac64_macosx.cocoa.x86_64_8.0.html

This test looks for these files in the plugins if they are not there we get this test failure
"about.html", "plugin.properties", "plugin.xml"

for source we look for about.html
Comment 48 Sravan Kumar Lakkimsetti CLA 2016-10-01 14:40:43 EDT
Created attachment 264525 [details]
Ant verbose log for generic editor testrun

Here is the piece of code where it is failing

  <target
    name="junit"
    depends="init"
    unless="skip.test">
    <antcall target="setup">
    </antcall>
    <property file="finalPluginsVersions.properties" />
    <property
      name="library-file"
      value="${executionDir}/library.xml" />
    <property
      name="junit-stylesheet"
      value="${executionDir}/JUNIT.XSL" />
    <echo>trying to find ${testPlugin}_*/test.xml</echo>
    <fileset
      id="test.plugin.file"
      dir="${eclipse-home}/plugins">
      <filename name="${testPlugin}_*/test.xml" />
    </fileset>
    <property
      name="testPluginX"
      refid="test.plugin.file" />
    <echo>trying to find ${testPluginX}</echo>
    <condition
      property="pluginexists"
      value="true">
      <not>
        <equals
          arg1="${testPluginX}"
          arg2="" />
      </not>
    </condition>
    <antcall target="runSuite" />
    <antcall target="genResults" />
  </target>

In case of generic editor testPluginX is getting the proper value it should get org.eclipse.ui.genericeditor.tests_1.0.0.N20160930-0900.jar/test.xml
Comment 49 Sravan Kumar Lakkimsetti CLA 2016-10-01 15:13:11 EDT
Found the problem. All test plugins have 

Eclipse-BundleShape: dir

property set in the manifest file. in case of generic editor this property is not set this is resulting in tests plugin to be installed as jar. not as extracted directory. 

Please make sure the tests plugin gets installed as extracted folder in plugins directory.
Comment 53 Dani Megert CLA 2016-10-03 09:08:52 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #52)
> Centos GTK3:
> https://hudson.eclipse.org/platform/job/ep47N-unit-cen64/ws/workarea/N20161002-2000/eclipse-testing/results/html/org.eclipse.ui.genericeditor.tests_ep47N-unit-cen64_linux.gtk.x86_64_8.0.html
> 

We have 1 error here, but at least the test now run.
Comment 54 Dani Megert CLA 2016-10-03 09:10:43 EDT
The Releng test no longer failed.
Comment 55 Dani Megert CLA 2016-10-03 12:14:28 EDT
(In reply to Dani Megert from comment #53)
> (In reply to Sravan Kumar Lakkimsetti from comment #52)
> > Centos GTK3:
> > https://hudson.eclipse.org/platform/job/ep47N-unit-cen64/ws/workarea/N20161002-2000/eclipse-testing/results/html/org.eclipse.ui.genericeditor.tests_ep47N-unit-cen64_linux.gtk.x86_64_8.0.html
> > 
> 
> We have 1 error here, but at least the test now run.

Sopot or Mickael, can you please investigate? That's the last item before we can mark this FIXED.
Comment 56 Mickael Istria CLA 2016-10-03 12:19:56 EDT
(In reply to Dani Megert from comment #55)
> Sopot or Mickael, can you please investigate? That's the last item before we
> can mark this FIXED.

We're on it, but since neither Sopot nor I can reproduce this issue, and since the Hudson slaves work for multiple environments and work for the Tycho tests, we're a bit clueless... I imagine that it could be a sync issue or some other popup showing at the same time.
Is there access to screenshots on such failure? Is the failure reproducible determinstically on the slave?
Comment 57 Nobody - feel free to take it CLA 2016-10-04 08:47:39 EDT
Status update: 

I tested with RHEL 7.2 which has GTK 3.14. My colleague tested for 3.16 and  3.18. Also tested on Fedora with 3.20. In all cases all tests succeed. Note that I'm testing by importing the test project and running from inside the IDE as a plug-in project. That is I'm not running the 'test.xml' way. Not sure if it makes a difference.

Now I'm getting CentOS 7 which should be almost identical to the one on which it's failing. Hopefully I'll reproduce it there.
Comment 58 Nobody - feel free to take it CLA 2016-10-05 08:27:57 EDT
Update:

I tested with stock CentOS 7 and all tests are green by running as JUnit plugin tests from the workspace.

Only thing left is for me to test by running the test framework. I asked Sravan for help as the guide in https://wiki.eclipse.org/Platform-releng/Automated_Testing is badly outdated. He gave me instructions to use https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/plain/production/testScripts/hudsonBootstrap/getEBuilder.xml 

Still the fact that there is no stable N-build is preventing me from using a build which has the tests in it. Right now I'm blocked until we have a stable build.
Comment 61 Eclipse Genie CLA 2016-10-07 09:24:02 EDT
New Gerrit change created: https://git.eclipse.org/r/82731
Comment 63 Nobody - feel free to take it CLA 2016-10-07 11:06:40 EDT
N&N entry also merged.

Dani, can you think of anything else left to do/check here?
Comment 64 Dani Megert CLA 2016-10-07 11:38:24 EDT
(In reply to Sopot Cela from comment #63)
> N&N entry also merged.
> 
> Dani, can you think of anything else left to do/check here?

I think we're done here as an initial step.
Comment 65 Mickael Istria CLA 2016-10-07 12:12:08 EDT
Awesome! Thanks to everyone involved! I believe this opens a new way of doing things and that the Generic Editor will reach its goal to unify addition of editing support just like the CNF unifies the navigation. Some good examples have already been produced with apparently much less effort than it would have been to produce dedicated editors.
Let's mark this one as resolved then.

For those interested in following up, here are some related bugs or enhancement requests to fix in generic and extensible editors
* Bug #500755
* Bug #504042
* Bug #503332
* Bug #505436
and some other related works
* Bug #505326
* Bug #500978
* Bug #500667
* Bug #496114 and https://github.com/eclipselabs/eclipse-language-service
* Bug #486961
Comment 66 Dani Megert CLA 2016-10-10 09:27:31 EDT
*** Bug 181907 has been marked as a duplicate of this bug. ***
Comment 67 Eclipse Genie CLA 2016-10-14 06:13:20 EDT
New Gerrit change created: https://git.eclipse.org/r/83198
Comment 69 Lars Vogel CLA 2016-10-26 11:44:33 EDT
(In reply to Eclipse Genie from comment #68)
> Gerrit change https://git.eclipse.org/r/83198 was merged to [master].
> Commit:
> http://git.eclipse.org/c/www.eclipse.org/eclipse/news.git/commit/
> ?id=544ffab6cde6c082cfcf27f9a46eff12c9c95475

Thanks Sopot for fixing the news contribution. As discussed, we should avoid using vendor specific packages names in the N&N.