Community
Participate
Working Groups
#0. The Request Allow the downstream client to register own logic to deal with special files like .project, .settings/, etc... when importing the projects. So the client can let them be generated into other places rather than the project root. (e.g. the workspace metadata folder). #1. Background A lot of VS Code Java (back-ended by Eclipse JDT.LS) complains that after importing Java projects, a lot of files generated in the root of each project, including: - .project - .settings/ - .classpath (this is generated by JDT actually) They hope these files not be generated at the project root since it will mess up their SCM. Here is the original issue, which is the most top one in the repo: https://github.com/redhat-developer/vscode-java/issues/618 #2. Findings & Potential Solutions Since the FileStore is used to provide the IO streams. We can consider to return a different FileStore(or FileStoreRoot) for some special files. Let it read/write content from/to the place we want. Below is a very simple snippet to illustrate the idea only, returning a special FileStoreRoot in FileSystemResourceManager: if (lastSegment.equals(IProjectDescription.DESCRIPTION_FILE_NAME)) { URI rootUri = workspace.getMetaArea().locationFor(target.getProject()).toFile().toURI(); IPath workspacePath = target.getProject().getFullPath(); return new FileStoreRoot(rootUri, workspacePath); } Then the FileStore created from this root can write files to the place we want to. In addition, because different languages may generate some special files at the project root as well. For example, JDT will generate .classpath for each Java project. We definitely don't want the Resource Plugin to handle such files since that belongs to the downstream project actually. So it could be great if Resource Plugin can provide a registration logic, that allows the downstream client to register its own logic to deal with those files according to its needs. How about: we extract those related part to an interface. (as Mickael Istria mentioned besides the FileSystemResourceManager.getFileRoot(), there are some other places we need to handle, like: FileSystemResourceManager.hasSavedDescription(), ...) The Resource Plugin has its default implementation of the interface, which is exactly the current existing code. Meanwhile, it exposes a way to allow the downstream client to register their own implementation of the interface. In that case, we won't break anything in Eclipse, and the downstream client is responsible for dealing with the interested files by their own.
> Since the FileStore is used to provide the IO streams. We can consider to > return a different FileStore(or FileStoreRoot) for some special files. Let > it read/write content from/to the place we want. > > Below is a very simple snippet to illustrate the idea only, returning a > special FileStoreRoot in FileSystemResourceManager: > > if (lastSegment.equals(IProjectDescription.DESCRIPTION_FILE_NAME)) { > URI rootUri = > workspace.getMetaArea().locationFor(target.getProject()).toFile().toURI(); > IPath workspacePath = target.getProject().getFullPath(); > return new FileStoreRoot(rootUri, workspacePath); > } > > Then the FileStore created from this root can write files to the place we > want to. I'm not sure it's the right approach. If I understand correctly, you're thinking about hacking FileStore so it intentionally returns "wrong" content for a given path (by wrong I mean returning a non-empty content for a non-existing file). I don't think this is the purpose of FileStore which is expected to properly map the underlying file resources. What you may want to look at is the various usage of IProjectDescription.DESCRIPTION_FILE_NAME. You'll find some in Project, or FileSystemResourceManager.internalWrite(). I imagine some preliminary refactoring would be to extract a "FileStore getProjectDescriptionFileStore()" method from Project, that would be reuse everywhere it makes sense when reading or writing the file. By using a FileStore here, and not a IFile, it will allow in a further step to put the file outside of the workspace directory, for example in a metadata location. Once this method is extracted, it would be easier to implement alternative strategies to load it, such as -if .project doesn't exist in folder and some magic setting is set- failing back to some known metadata location. > In addition, because different languages may generate some special files at > the project root as well. For example, JDT will generate .classpath for each > Java project. We definitely don't want the Resource Plugin to handle such > files since that belongs to the downstream project actually. So it could be > great if Resource Plugin can provide a registration logic, that allows the > downstream client to register its own logic to deal with those files > according to its needs. I think such tweaks for .classpath/ location should be implemented in JDT. > How about: we extract those related part to an interface. (as Mickael Istria > mentioned besides the FileSystemResourceManager.getFileRoot(), there are > some other places we need to handle, like: > FileSystemResourceManager.hasSavedDescription(), ...) The Resource Plugin > has its default implementation of the interface, which is exactly the > current existing code. Meanwhile, it exposes a way to allow the downstream > client to register their own implementation of the interface. In that case, > we won't break anything in Eclipse, and the downstream client is responsible > for dealing with the interested files by their own. I'm not sure an interface is necessary; as I don't think we should at the moment assume we want extensibility here. What's been requested is an alternative strategy that uses workspace metadata area to store the .project (and the .settings/ but it's also a different plugin and solution I believe), so we should stick to it and implement it, without introducing extra extensibility or complexity. Once there is agreement the alternative strategy can be and how we enable it, then we can just code it in the plugin. IMO, it would lead to better and faster results. Concretely, once we have extracted a getProjectDescriptionFileStore() method from Project and are using it read/write .project files; then we can imagine having some workspace preference to decide how to proceed if no such .project file exists: lookup/create it locally or in metadata.
(In reply to Mickael Istria from comment #1) > Once there is agreement the alternative strategy can be and how we enable > it, then we can just code it in the plugin. IMO, it would lead to better and > faster results. Thank you Mickael! Yes I really hope that this issue can be somehow addressed ASAP (and be nicely of cause :D). > > > Since the FileStore is used to provide the IO streams. We can consider to > > return a different FileStore(or FileStoreRoot) for some special files. Let > > it read/write content from/to the place we want. > > > > Below is a very simple snippet to illustrate the idea only, returning a > > special FileStoreRoot in FileSystemResourceManager: > > > > if (lastSegment.equals(IProjectDescription.DESCRIPTION_FILE_NAME)) { > > URI rootUri = > > workspace.getMetaArea().locationFor(target.getProject()).toFile().toURI(); > > IPath workspacePath = target.getProject().getFullPath(); > > return new FileStoreRoot(rootUri, workspacePath); > > } > > > > Then the FileStore created from this root can write files to the place we > > want to. > > I'm not sure it's the right approach. If I understand correctly, you're > thinking about hacking FileStore so it intentionally returns "wrong" content > for a given path (by wrong I mean returning a non-empty content for a > non-existing file). I don't think this is the purpose of FileStore which is > expected to properly map the underlying file resources. Hmm... Maybe I didn't explain the idea clearly. Yes FileStore is used to map the IResource in the workspace tree to the real file in the file system. What I want to do is given the resource: '/<Project>/.project'. Now the file store maps this IPath to the file located in meta data area instead of the at the project root. The above code maps the project's IPath to a path in the matadata area. Then when calling fileStoreRoot.createStore(), the relative path (based on the project's IPath) will simply append to it. See LocalFile.getFileStore(). > I think such tweaks for .classpath/ location should be implemented in JDT. I'm still new to the Platform.Resources project. From my current understanding, write or read a resource in a workspace tree finally relies on the FileStore of that IResource. JDT uses IFile.setContents(), IFile.create() to write .classpath file. I'm not sure if it is possible for JDT to change the behavior since it's impossible for it to change the underlying FileStore I guess?
I think we don't need to, and shouldn't, hack a FileStore here. I'm afraid there are actually invariants built on the idea that the output of filestore is a regular tree under the hood. However, there is already a solution to this issue for JDT or other files, that could be leverages by those projects: linked resources https://help.eclipse.org/latest/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2FresInt_linked.htm . Those have been available for a long time and should be reliable enough. Any IFolder (eg .settings/) or IFile (eg .classpath) created by some downstream project could be created with createLink instead of regular create and then they'd be free to use the location they want and everything else would work as usual on the consumer side: files would be visible in the UI (unless marked as HIDDEN), editable and so on (so it can be convenient to hack things sometimes), but they'd be stored in another location (such as workspace metadata) and shouldn't mess up with Git. The only special file from Resource model is the .project as it's a driver for Project object creation upon restart; but I believe it's much more proper and less risky to implement an explicit solution for it in Project than to hack the underlying FileStore. I think you can already open bugs to JDT for .classpath and a separate bug to Resources for .settings/ about allowing them to be created as links; and we should keep focus on .project here.
(In reply to Mickael Istria from comment #3) Sure I'll file a bug to JDT to start the discussion first. While, my concerning for the link resource proposal is that, if there are some other plugins already hard coded the path in their code, then it will break, right?
Downstream plugins are supposed to use APIs, models, services provided by Platform. The physical location of the resources on filesystem is not part of any API contract as far as I remember, so it's not an issue to change that.
Beware when using linked resources: Eclipse has a preference to forbid creation of linked resources (ResourcesPlugin.PREF_DISABLE_LINKING), and there's a UI for it (Preferences->General->Workspace->Linked Resources). If anything is done here, it must also work if the user (or the product) disables linked resources. Also note that other plug-ins may currently assume that .project is in the project's root folder. For instance importing existing projects. I don't see how that could work if the file was stored somewhere else. IMO the correct way to deal with these IDE-specific files is indeed to have them inside the project's root directory, and to check them in in SCM. That way anyone checking them out gets the correct setup.
(In reply to Thomas Wolf from comment #6) > Beware when using linked resources: Eclipse has a preference to forbid > creation of linked resources (ResourcesPlugin.PREF_DISABLE_LINKING), and > there's a UI for it (Preferences->General->Workspace->Linked Resources). If > anything is done here, it must also work if the user (or the product) > disables linked resources. > > Also note that other plug-ins may currently assume that .project is in the > project's root folder. For instance importing existing projects. I don't see > how that could work if the file was stored somewhere else. The context here us JDT-LS, which is a dedicated RCP application that would decide of the preferences for the user and would implement is own import. So the worries here wouldn't really apply. There is no need to change the default in Eclipse IDE, and if there is some further development happening,it would mostly be for project creation/import and the related preference could be checked first. > IMO the correct way to deal with these IDE-specific files is indeed to have > them inside the project's root directory, and to check them in in SCM. That > way anyone checking them out gets the correct setup. That's indeed what works best, but it's unfortunately not what most users want, and it's worth being addressed in the resource model.
(In reply to Mickael Istria from comment #5) I think one benefit of using the extension point is that, it allows different clients to have different opinions on where these generated files should be persisted. For VS Code Java, we hope they can be generated under the workspace metadata folder. But this might not be applied to other clients like Eclipse Che/Eclipse Theia. During the investigation, I found the FileStore/FileStoreRoot is a great entry point because it holds the file's IO stream. I did a POC and found it works well, the code change is small. Of cause I totally understand your concern. If directly dealing with FileStore/FileStoreRoot looks too hacky. Will it be possible that we abstract it to a higher level. For example, we allow the client has their own implementation of the workspace, or maybe FileSystemResourceManager? // BTW, I've filed a ticket at JDT side to ask about the linked resource approach: https://bugs.eclipse.org/bugs/show_bug.cgi?id=576175. We can keep the discussion parallelly.
(In reply to Sheng Chen from comment #8) > I think one benefit of using the extension point is that, it allows > different clients to have different opinions on where these generated files > should be persisted. If using the filesystems extension point works for you, then.just use it. But I don't think we'll allow a new extension point for this standard case. > For VS Code Java, we hope they can be generated under the workspace metadata > folder. But this might not be applied to other clients like Eclipse > Che/Eclipse Theia. Is that certain? What are their specific requirements? I.just want to make sure we design the simplest and nest and most reusable solution for.real problems, not that we rely on too much hypothesis to drive the design. > During the investigation, I found the FileStore/FileStoreRoot is a great > entry point because it holds the file's IO stream. I did a POC and found it > works well, the code change is small. As mentioned earlier, I disagree it's the right layer and while it may work, it's not giving responsibility of this functional change where it belongs (in Project). > Of cause I totally understand your concern. If directly dealing with > FileStore/FileStoreRoot looks too hacky. Will it be possible that we > abstract it to a higher level. For example, we allow the client has their > own implementation of the workspace, or maybe FileSystemResourceManager? Have you tried what I suggested earlier aout preparing the Project code to.read/write project data to alternative filestore? What makes you think therewould be need for alternative strategies the the current one (used for 21 years and dozens million users) and the one you suggest (requested also several times in the past)? If we currently have only those 2 implementations to consider, they'd better be harcoded internally for the moment with a preference to decide which on to use when creating a new project,and maybe later, if more extensibility seems necessary and 3rd strategy seems necessary for some product but not desired inside the Platform, we'll consider Opening to extensibility.
Cycling back to the 1st steps, I think the request expressed here is just one possible solution of the functional need, and wouldn't either be the simplest nor the most profitable. I think we should retitle the scope of the issue according to your actual functional need; with something like "allow storing project metadata in workspace metadata folder". This case is important enough and has been requested enough times without much variation to be part of Platform without messing around with extensibility.
(In reply to Mickael Istria from comment #10) > Cycling back to the 1st steps, I think the request expressed here is just > one possible solution of the functional need, and wouldn't either be the > simplest nor the most profitable. > > I think we should retitle the scope of the issue according to your actual > functional need; with something like "allow storing project metadata in > workspace metadata folder". This case is important enough and has been > requested enough times without much variation to be part of Platform without > messing around with extensibility. Thanks for the suggestion. I just re-titled this bug. Meanwhile, I can take a look at the filesystems extension point to see if there is any possibility there.
Hi Mickael, One question about the filesystems extension point. Is it overridable if another plugin also registered the 'file' scheme and be loaded?
(In reply to Sheng Chen from comment #12) > Hi Mickael, > > One question about the filesystems extension point. Is it overridable if > another plugin also registered the 'file' scheme and be loaded? Forget about it, I just tried and the answer is yes. This extension point seems promising!
(In reply to Sheng Chen from comment #13) > (In reply to Sheng Chen from comment #12) > > Hi Mickael, > > > > One question about the filesystems extension point. Is it overridable if > > another plugin also registered the 'file' scheme and be loaded? > > Forget about it, I just tried and the answer is yes. This extension point > seems promising! I'm surprised it is overridable, and I think it should be investigated whether this is reliable and not "random" (ie some strange criteria happen under the hood and may select an extension or the other according to external settings your application won't be able to manage). Anyway, good for you if it's working, please share a note about it here when you have some code that can be referred to as example.
@Sheng Chen please ensure that your solution will not block the alternative filesystem implementation from being activated for particular project. We are using this extension point to hide the actual file location from user in order to block manual/external modification for project resources.
(In reply to Mickael Istria from comment #14) > I'm surprised it is overridable, and I think it should be investigated > whether this is reliable and not "random" (ie some strange criteria happen > under the hood and may select an extension or the other according to > external settings your application won't be able to manage). From what I see, it's because the bundle of JDT.LS is always loaded after the resources plugin. Thus the extension of JDT.LS will always have a larger id and override the extension of the resource plugin (in the for-loop of InternalFileSystemCore#getFileSystemRegistery()). That means the extension of JDT.LS can be overridden as well. (But it's not a problem to concern for now I think) > Anyway, good for you if it's working, please share a note about it here when > you have some code that can be referred to as example. Sure I will do that. (We will have our National holiday so I'll share my findings in next month when I back to office :) )
(In reply to Sheng Chen from comment #16) > > From what I see, it's because the bundle of JDT.LS is always loaded after > the resources plugin. Thus the extension of JDT.LS will always have a larger > id and override the extension of the resource plugin (in the for-loop of > InternalFileSystemCore#getFileSystemRegistery()). > > That means the extension of JDT.LS can be overridden as well. (But it's not > a problem to concern for now I think) > I hope you are not going to rely on the order of extension reading.
(In reply to Alexander Fedorov from comment #17) > I hope you are not going to rely on the order of extension reading. Yes that's a problem I must concern to make sure the feature is stable. But, anyway this extension point is still worth trying. I'm making a POC now and will share my findings here.
I just came back from the vacation and have some updates on this. I did following things to experiment the POC: 1. Register a new file system provider to the extension point org.eclipse.core.filesystem.filesystems 2. In implementation, extends org.eclipse.core.internal.filesystem.local.LocalFileSystem and override some methods: public class JdtlsFileSystem extends LocalFileSystem { @Override public IFileStore getStore(IPath path) { ... } @Override public IFileStore getStore(URI uri) { ... } } 3. The FileStore returned from the JdtlsFileSystem is a class which extends org.eclipse.core.internal.filesystem.local.LocalFile: public class JdtlsFile extends LocalFile { public JdtlsFile(File file) { super(file); } @Override public String[] childNames(int options, IProgressMonitor monitor) { // because metadata files are stored in workspace, so we need to have some sepcial logic to make sure refreshLocal() works correctly. ... } @Override public IFileStore getChild(String name) { ... } @Override public IFileStore getFileStore(IPath path) { ... } } Basically, the overriden methods will return a file store which locates in workspace when the target resource is a project metadata file. So far it works well in most cases, but I did meet some problems: 1. Calling Resource.getLocation() & Resource.getRawLocation() will return a location at the project root. This is because in FileStoreRoot#localLocation(), it appends the relative path to the root path directly. (getLocationURI() works because FileStoreRoot#computeURI() will call EFS.getStore(rootURI).getFileStore(childPath) to get the result). 2. Gradle project still generates .settings/ at the project root. After looking into the implementation of Buildship, I found that this is because the path is hard-coded in the code: https://github.com/eclipse/buildship/issues/1111. So this should be a separate issue from this one. For Problem 1, is there any suggestions?
IMO it works correctly for Problem 1 since this extension point was designed to support things like remote file system and not to provide alternative location for particular resource like `.project`. I still think that alternative strategy to store project metadata needs dedicated extension point since it is orthogonal to the functionality of "org.eclipse.core.filesystem.filesystems"
(In reply to Alexander Fedorov from comment #20) > I still think that alternative strategy to store project metadata needs > dedicated extension point since it is orthogonal to the functionality of > "org.eclipse.core.filesystem.filesystems" No extension point is necessary. It's "standard" and there is no obvious variability in the requirements to make a feature/option of the core Resource API.
Is it suggested to solve this issue as what we've discussed at beginning: solve it piece by piece and start from the .project file?
(In reply to Mickael Istria from comment #21) > (In reply to Alexander Fedorov from comment #20) > > I still think that alternative strategy to store project metadata needs > > dedicated extension point since it is orthogonal to the functionality of > > "org.eclipse.core.filesystem.filesystems" > > No extension point is necessary. It's "standard" and there is no obvious > variability in the requirements to make a feature/option of the core > Resource API. How this approach could be combined with really _standard_ "org.eclipse.core.filesystem.filesystems" functionality, i.e. how to configure .project to use non-default location for remote file system?
(In reply to Alexander Fedorov from comment #23) > How this approach could be combined with really _standard_ > "org.eclipse.core.filesystem.filesystems" functionality, i.e. how to > configure .project to use non-default location for remote file system? See comment #1.
(In reply to Sheng Chen from comment #22) > Is it suggested to solve this issue as what we've discussed at beginning: > solve it piece by piece and start from the .project file? It depends on how much effort you want to place in this work. The current "hack" using org.eclipse.core.filesystem.filesystems seems to work decently already for you, despite a few glitches. If it's acceptable for your case, I'd recommend you stick to it as long as it works (make sure to cover your needs well with automated tests as it's prone to regression - eg relying on the extensions order which I believe is not specified). But if you want a "proper" implementation, I think it should indeed be what I already suggested in comment #1, with adding into the actual Workspce model that capability to start having a way to read .project files that are inside workspace metadata. It's not trivial but some preliminary steps can already be implemented safely and with a positive impact on the codebase anyway. The .project file is both the start and the end; other issues you face (eg JDT config files, Gradle forcing .project at a given location...) would have to be fixed in the respective plugins by allowing usage of linked resources or other options at import.
(In reply to Mickael Istria from comment #25) > (In reply to Sheng Chen from comment #22) > > Is it suggested to solve this issue as what we've discussed at beginning: > > solve it piece by piece and start from the .project file? > > It depends on how much effort you want to place in this work. The current > "hack" using org.eclipse.core.filesystem.filesystems seems to work decently > already for you, despite a few glitches. If it's acceptable for your case, > I'd recommend you stick to it as long as it works (make sure to cover your > needs well with automated tests as it's prone to regression - eg relying on > the extensions order which I believe is not specified). > But if you want a "proper" implementation, I think it should indeed be what > I already suggested in comment #1, with adding into the actual Workspce > model that capability to start having a way to read .project files that are > inside workspace metadata. It's not trivial but some preliminary steps can > already be implemented safely and with a positive impact on the codebase > anyway. > The .project file is both the start and the end; other issues you face (eg > JDT config files, Gradle forcing .project at a given location...) would have > to be fixed in the respective plugins by allowing usage of linked resources > or other options at import. My main concern regarding this "hack" is that it can influence others when installed in the same Eclipse instance via "shadowing" the LocalFileSystem implementation.
(In reply to Alexander Fedorov from comment #26) > My main concern regarding this "hack" is that it can influence others when > installed in the same Eclipse instance via "shadowing" the LocalFileSystem > implementation. I don't see a risk of this being broken by improving the Workspace directly because * storing .project files in workspace metadata instead of under project root would be and opt-in capability, so by default that wouldn't change. * the workspace model is a layer above the filesystem model; so if we make the workspace capable of using different IPath/location/uri for .project, then the underlying FileSystem implementation would still be queried in the end and customizations in place would still be performed as defined in the custom FileSystem.
(In reply to Mickael Istria from comment #25) > It depends on how much effort you want to place in this work. The current > "hack" using org.eclipse.core.filesystem.filesystems seems to work decently > already for you, despite a few glitches. If it's acceptable for your case, > I'd recommend you stick to it as long as it works (make sure to cover your > needs well with automated tests as it's prone to regression - eg relying on > the extensions order which I believe is not specified). > But if you want a "proper" implementation, I think it should indeed be what > I already suggested in comment #1, with adding into the actual Workspce > model that capability to start having a way to read .project files that are > inside workspace metadata. It's not trivial but some preliminary steps can > already be implemented safely and with a positive impact on the codebase > anyway. > The .project file is both the start and the end; other issues you face (eg > JDT config files, Gradle forcing .project at a given location...) would have > to be fixed in the respective plugins by allowing usage of linked resources > or other options at import. Thanks for the suggestion. Yes I agree for long term consideration, we should address this issue more properly. And this can be done parallelly with the current approach. For sure, the other issues like buildship one should be addressed in buildship. I'll keep testing the current approach to check if it really meets the requirement and share new findings if I have.
Some update: Sorry for the late update. We finally take the File System Provider extension point since that is currently the best solution we know for this problem. (quick to implement and almostly transparent to other modules). Details for the change can be found here: https://github.com/eclipse/eclipse.jdt.ls/tree/master/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/filesystem The fixed was released around 1 week ago, so far it looks good. Thank you Mickael and Alexander for your great help and suggestions. That issue is the most voted one in our repo and it won't be addressed so quickly without your help!
