Bug 480551 - Enable "refresh using native hooks"
Summary: Enable "refresh using native hooks"
Status: CLOSED MOVED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Architecture Council (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal with 1 vote (vote)
Target Milestone: ---   Edit
Assignee: eclipse.org-architecture-council CLA
QA Contact:
URL:
Whiteboard: "stalebug"
Keywords: feep
Depends on:
Blocks:
 
Reported: 2015-10-23 21:14 EDT by Pascal Rapicault CLA
Modified: 2021-12-23 06:33 EST (History)
14 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Pascal Rapicault CLA 2015-10-23 21:14:30 EDT
The number one complaint on ihateeclipse.com is about Eclipse not automatically refreshing files. Back then in 2001, it made sense not enable that option by default, but now that Eclipse requires Java 8 and that the JRE provides a file polling API, I don't see a reason not to fix this #1 issue.
Comment 1 Marcel Bruch CLA 2015-10-24 02:36:37 EDT
I can support this proposal proposal with the observation in the error reports. A significant share of the error reports are caused by FileNotFoundExceptions. Many (if not most) of them are caused by an outdated workspace model of the filesystem.

A secondary observation:
Some plugins (e.g. JGit) rely on the java.io.File APIs directly and thus are causing many, long UI freezes. It may make sense to to reconsider the implementation *if* the new File APIs offer a better performance (but I don't know)

+1 for the general proposal b/c I think it will reduce the number of error reports and thus will be recognized by users.
Comment 2 Pascal Rapicault CLA 2015-10-24 21:08:04 EDT
> A secondary observation:
> Some plugins (e.g. JGit) rely on the java.io.File APIs directly and thus are
> causing many, long UI freezes. It may make sense to to reconsider the
> implementation *if* the new File APIs offer a better performance (but I
> don't know)
   Could you be more specific on this? Are you indicating that JGit usage of the polling API is causing UI freezes?
   Should we spend FEEP money on EGit?
Comment 3 Dani Megert CLA 2015-10-26 06:46:03 EDT
This is about changing a preference, i.e. a few minutes of work. I don't see why this needs any funding.
Comment 4 Pascal Rapicault CLA 2015-10-26 11:54:10 EDT
FEEP because I was not sure that we had the right plug-ins for each platform, and I was not sure that they were fully functional and tested.
Comment 5 Dani Megert CLA 2015-10-26 12:03:13 EDT
(In reply to Pascal Rapicault from comment #4)
> FEEP because I was not sure that we had the right plug-ins for each
> platform, and I was not sure that they were fully functional and tested.

Then the summary is wrong - you are requesting to have the native hooks for all primary platforms. IIRC we now use the platform independent Java 7 IO stuff (org.eclipse.core.filesystem.java7). Szymon, please confirm or clarify.
Comment 6 Szymon Ptaszkiewicz CLA 2015-10-26 12:26:50 EDT
(In reply to Dani Megert from comment #5)
> (In reply to Pascal Rapicault from comment #4)
> > FEEP because I was not sure that we had the right plug-ins for each
> > platform, and I was not sure that they were fully functional and tested.
> 
> Then the summary is wrong - you are requesting to have the native hooks for
> all primary platforms. IIRC we now use the platform independent Java 7 IO
> stuff (org.eclipse.core.filesystem.java7). Szymon, please confirm or clarify.

Indeed, the summary does not match the description. To clarify:

1. Native hooks for Windows exist inside org.eclipse.core.resources.win32.x86.
2. For Linux we have bug 108697.
3. The org.eclipse.core.filesystem.java7 plugin is not about native hooks but about having implementation of org.eclipse.core.filesystem.IFileInfo that does not require any native code. We want to have/keep native implementation of IFileInfo because it is faster than when using Java 7 API, thus Java 7 API is used only as a backup solution on platforms where there is no native implementation of IFileInfo.
4. By "native hooks" we usually mean a callback-style of notification that can inform Eclipse that something has changed on the filesystem (like in the observer pattern). Traditionally, it was always platform specific as the "native" suggests (see points 1 and 2 above) but based on bug 108697 comment 8 it seems to be doable with Java 7 API.
Comment 7 Szymon Ptaszkiewicz CLA 2015-10-26 12:42:47 EDT
(In reply to Marcel Bruch from comment #1)
> I can support this proposal proposal with the observation in the error
> reports. A significant share of the error reports are caused by
> FileNotFoundExceptions. Many (if not most) of them are caused by an outdated
> workspace model of the filesystem.

I don't think it is related because as far as I saw in the error reports, problems with "outdated workspace model" due to external changes on the filesystem happen equally on Windows and Linux while on Windows we do have native hooks. I am not sure if the native hooks option is enabled or not by default in EPP, but if enabling it does not reduce the number of reported problems with "outdated workspace model" on Windows, then I don't think it will justify putting effort in implementing native hooks for other platforms, at least from the number of reported errors perspective.
Comment 8 Pascal Rapicault CLA 2015-10-26 13:31:00 EDT
> Then the summary is wrong - you are requesting to have the native hooks for
> all primary platforms. IIRC we now use the platform independent Java 7 IO
> stuff (org.eclipse.core.filesystem.java7). Szymon, please confirm or clarify.

My request is for the errors reported and the complaints filed to go away. If you think it is enough to have the preferences enabled, then so be it. 

I may have wrongly assumed that enabling this option would require a native fragment on all platforms. If it is not the case great. So question, what happens when the option is enabled the fragments are not available? Do we poll the filesystem?
Comment 9 Pascal Rapicault CLA 2015-10-26 13:33:03 EDT
> I don't think it is related because as far as I saw in the error reports,
> problems with "outdated workspace model" due to external changes on the
> filesystem happen equally on Windows and Linux while on Windows we do have
> native hooks. 
   I downloaded Java package SR1 and "Refresh using native hooks" is *not* enabled. "Refresh on access" is enabled.
Comment 10 Szymon Ptaszkiewicz CLA 2015-10-28 08:15:05 EDT
(In reply to Pascal Rapicault from comment #8)
> So question, what happens when the option is enabled the fragments are not
> available? Do we poll the filesystem?

Yes. The Javadoc of the ResourcesPlugin.PREF_AUTO_REFRESH preferences which is controlled by the "Refresh using native hooks or polling" option says:

"Auto-refresh installs a file-system listener, or performs periodic file-system polling to actively discover changes in the resource hierarchy."
Comment 11 Michael Scharf CLA 2015-10-29 07:31:01 EDT
> "Auto-refresh installs a file-system listener, or performs periodic
> file-system polling to actively discover changes in the resource hierarchy."

I think we have to check what it actually does on the systems supported by eclipse. It could well be that it uses a file-system listener for windows, linux and mac.
Comment 12 Thomas Schindl CLA 2015-11-18 10:28:00 EST
Just in case it matter NIO-File watching on OS-X is extremly slow and its not sure if this gets fixed with Java7 - I wrote my custom implementation who uses the OS-X low-level APIs.
Comment 13 Mikaël Barbero CLA 2016-03-03 10:07:26 EST
(In reply to Thomas Schindl from comment #12)
> Just in case it matter NIO-File watching on OS-X is extremly slow and its
> not sure if this gets fixed with Java7 - I wrote my custom implementation
> who uses the OS-X low-level APIs.

Is this custom implementation available in OSS? Is it something close to https://github.com/gjoseph/BarbaryWatchService?
Comment 14 Thomas Schindl CLA 2016-03-03 10:35:36 EST
(In reply to Mikael Barbero from comment #13)
> (In reply to Thomas Schindl from comment #12)
> > Just in case it matter NIO-File watching on OS-X is extremly slow and its
> > not sure if this gets fixed with Java7 - I wrote my custom implementation
> > who uses the OS-X low-level APIs.
> 
> Is this custom implementation available in OSS? Is it something close to

no

> https://github.com/gjoseph/BarbaryWatchService?

yes similar but i did not implement the java-nio API
Comment 15 Mikaël Barbero CLA 2016-03-03 11:06:03 EST
(In reply to Thomas Schindl from comment #14)
> > https://github.com/gjoseph/BarbaryWatchService?
> 
> yes similar but i did not implement the java-nio API

He did not either. He just named his API similarly ;)
Comment 16 Mikaël Barbero CLA 2016-03-04 03:29:27 EST
(In reply to Dani Megert from comment #3)
> This is about changing a preference, i.e. a few minutes of work. I don't see
> why this needs any funding.

Agreed. I would better see bug 108697 to be solved under FEEP. I've added bug 108697 comment 12 to start the discussion about a refresh monitor based on Java7+ WatchService. 

Once bug 108697 is solved, this bug should also be addressed. But as you said, it's a few minutes of work.
Comment 17 Brian de Alwis CLA 2016-03-31 15:29:36 EDT
FYI: I've been running with an FSEvents-based auto-refresher [1] and started experiencing hangs on startup.  It seems the FSEvents initialization may interfere play well with SWT.

[1] http://jonatan.virtuallypreinstalled.com/autorefresher
Comment 18 Mikaël Barbero CLA 2016-04-11 05:13:53 EDT
(In reply to Brian de Alwis from comment #17)
> FYI: I've been running with an FSEvents-based auto-refresher [1] and started
> experiencing hangs on startup.  It seems the FSEvents initialization may
> interfere play well with SWT.
> 
> [1] http://jonatan.virtuallypreinstalled.com/autorefresher

Can't access the link content. Could you please check it is correct? Thanks.
Comment 19 Brian de Alwis CLA 2016-04-11 10:11:58 EDT
That's the p2 site.

Marketplace entry: https://marketplace.eclipse.org/content/autorefresher
Github: https://github.com/zch/AutoRefresher/
https://vaadin.com/directory#!addon/autorefresher

I realize I should have filed an issue on the Github repo.
Comment 20 Mikaël Barbero CLA 2016-05-24 17:36:50 EDT
Thanks Brian for the links. I've started to experiment implementing my own FSEvents based auto refresher. So far, I don't face any interference with SWT. In the code you've referenced, I think the JNI code does not respect all best practices (no release of copied/pinned objects, mix of native threads and vm threads) which can all cause a lot of issues. I'm looking forward for your feedback on my implementation as soon as I have an publishable version.
Comment 21 Mikaël Barbero CLA 2016-06-01 14:25:31 EDT
(In reply to Mikaël Barbero from comment #20)
> Thanks Brian for the links. I've started to experiment implementing my own
> FSEvents based auto refresher. So far, I don't face any interference with
> SWT. In the code you've referenced, I think the JNI code does not respect
> all best practices (no release of copied/pinned objects, mix of native
> threads and vm threads) which can all cause a lot of issues. I'm looking
> forward for your feedback on my implementation as soon as I have an
> publishable version.

Available in https://git.eclipse.org/r/74291.
Comment 22 Eclipse Genie CLA 2018-11-13 16:38:25 EST
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.
Comment 23 Eclipse Genie CLA 2020-11-03 07:43:20 EST
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.
Comment 24 Frederic Gurr CLA 2021-12-23 06:33:14 EST
This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/218.