Community
Participate
Working Groups
Currently in Eclipse there is a direct connection between IResources and files and directories on the local file system. Eclipse should loosen this connection, by abstracting out its dependency on java.io.File, and allowing for alternative implementations. This would enable, for example, uses where the workspace is located on a remote server, or accessed via a non-file-based API, or has a non-trivial mapping between the resources and the physical layout of the files. [Platform Resources, Text]
I'm guessing this is somehow related to the work on Bug 37723? This is going to be very exiting; been waiting for non-physical file support for years.
Morten (and anyone else CC'd on the bug), can you briefly describe what you are looking for in this support? Are you interested in supporting remote file systems, some other storage such as a local database, or some transformation of a local file system? Do you want full flown IDE functionality on these files, or are you on the RCP level?
Sure I can give it a shot: I'm interested in the platform level mostly; not the Java IDE level (although I'd like to see the ability to have projects remotely over ftp and similar like most other IDEs can). The main reason I'd like "logical" resources is so that our application (which the whole repository is database backed) can participate in the common Resource views like "Problems", "Tasks" etc (close to what Mike Wilson described in Bug 37723 Comment 31). Marker and Annotation support is also in the same area. The simplest way of doing this would be to be able to implement my own IResource that isn't file based. We're currently working around the Marker issues (we use xml/javascript editors against non-filebased data) by making own Annotation store (top of my head I don't exactly remember how that is done), but we cannot participate in the Problems view for these (or at least I haven't found a way). -- There is also the issue that when data is remote, most third party plug-ins gets hard to work with because they all[most] rely on IFileEditorInput, but that problem probably wouldn't go away with this bug right away. In the end, I'd like it to be as easy as IntelliJ or Netbeans to define remote stores/projects. If I remember right, creating a custom file/data store in IntelliJ wasn't more than a couple of hundred lines of code.
Just to come with more wishes. Some are probably obvious :) : -- If we do get remote IResouces (or something similar), I'd like to be able to trap copying or moving IResources that come from my store if someone tries to copy it to another store (like copy from my database to the file system). I'd like to be able to support that, but the data might be a little different (maybe even allow to open up some wizard or similar). A simple usecase could be that I made a IMAP IResource store to read email and I want to be able to copy the email or attachments into the file system by copy/pasting. But the content of the IResource might be different when I want it on the filesystem compared to in memory from the IMAP server (for efficiency reasons). -- Not sure how, but I'd like the remote servers to control/serve markers. For example, if we're in a collaborative system, I might want "Tasks" annotated to a remote IResource to be shared between everyone against that remote server. The server should take care of ownership and security I guess. I believe such things are currently only stored/handled in the workspace. -- This might be unrelated, but I'd like to say that a "project" uses a particular "remote store" (maybe even project types). This way instead of having our own "Resource view" like we do today, I can use Project natures and remote IResources and integrate much better with Eclipse.
See also bug 109194.
*** Bug 109476 has been marked as a duplicate of this bug. ***
As part of this plan item, I have prototyped an abstract file system API. This API provides an extension point to allow other plugins to provide implementations for other file systems. Clients of this API use this abstraction layer as a replacement for interaction with java.io.File. An initial cut of this API has been released into a new plugin called "org.eclipse.core.filesystem", available now in dev.eclipse.org/cvsroot/eclipse. This plugin also provides an implementation for interaction with the local file system. I have ported the org.eclipse.core.resources plugin to run on top of this abstract API, and this work is currently in a CVS branch called "v20050705_FileSystemAPI" (for org.eclipse.core.resources and org.eclipse.core.tests.resources). Coments, feedback, and criticism of this API is most welcome, either in this bug report or on the platform-core-dev mailing list. In particular, I would be very interested to hear from people who have potential uses for this API as clients, and from those who want to plugin different file system implementations. While I have prototyped both local file system and an FTP-based file system, the API could benefit from a few other file systems implementations to validate that the abstraction layer is sufficient for a broad enough set of different file systems.
Created attachment 27466 [details] Team project set for FTP sample file system I have had questions about the FTP file system implementation. It is currently a crude prototype that I built for testing purposes. If you want to browse the code, import this team project set into your workspace. It currently takes a bit of hacking of the new project wizard to get it to accept an FTP URL as the project location. I am still working on getting the UI sorted out...
Do you know that there is a somewhat similar work being done at the Jakarta Commons project? It is called VFS and right now it's in the incubation phase: http://jakarta.apache.org/commons/sandbox/vfs/ It seems pretty elaborate as I myself am experimenting with it for an Eclipse RCP based file manager. This library might already provide what you guys need. It just needs a nicer Eclipse based packaging of plug-ins. This can be done by implementing a custom FileSystemManager which uses your extension point to configure the virtual file system capabilities. I think it's a nicer approach than to create unnecessary dependencies to Eclipse for a generic virtual file system library.
VFS looks nice on paper. For example being able to support Webdev would be very nice. And not having to implement all these 'file systems' again in Eclipse would be nice too.
I was wondering if you found many plug-ins that assumed that IResource.getLocation().toFile() could fail even if the resource existed. Also, you didn't answer the comments about how this refactoring affects the following IResource features: - markers: can you create markers on these new IResource implementations? - builders: how are they affected and do they know about slow/fast file systems? - change listeners: are they affected? - copy/delete/rename: can the IResource implementation override this behavior?
Ed, thanks for the pointer. We already moved VFS out of sandbox and its located on http://jakarta.apache.org/commons/vfs/ now. I the next few days I will create a new RC and hope we can vote on it a 1.0 release soon. I think it would be nice if we manage to use VFS for your needs, if there is anything I can do (e.g fix bugs ;-) ) feel free to contact me.
I'll answer comment #11 first because it's easier: - Clients calling IResource.getLocation().toFile() will continue to function as always for the cases where the file is still in the local file system. In other cases, they will get an exception. If clients want to function against arbitrarily backed file systems they need to stop using getLocation(). In all the cases I have found, there is an alternative to getLocation() that will work for non-local files. - markers: No change to markers. Essentially, this work is about storing file contents elsewhere - everything else about resources is completely unchanged. - builders: Only affected, as with other clients, if they assume resources are stored in the local file system (via calls like getLocation()toFile()). - change listeners: No change. - copy/delete/rename: The move/delete hook remains for repositories that want to override the default behaviour.
We did seriously evaluate apache VFS. The main thing it lacks is support in the API for progress reporting and cancelation. Especially for the cases where file contents are accessed over a network, it is essential that the user be able to cancel long running work and be notified of progress. I also didn't see any way to carry over some important optimizations we have made in Eclipse for file system access. Native I/O libraries typically return a struct containing all the relevant info about that file (FindFirstFile on Windows, stat() on *nix machines). This includes whether the file exists, its attributes, its last modified time, and other information. java.io.File, and FileObject in VFS, don't seem to provide API to get all that information in a single I/O call. Thus multiple JNI and native I/O calls are required to obtain information that could be available with a single call. We used our own native libraries to optimize this in Eclipse, and the proposed FileStore provides API to get all this data in a single call. These optimizations brought us some significant performance gains, and we really don't want to take a step backward and move to an API that doesn't provide something comparable. These optimization become even more significant in an environment where the file system is accessed over a network. Finally, the local file implementation of VFS doesn't seem to support setting attributes such as read-only, executable, archive, hidden, etc, or even reading some of those attributes (such as the executable attribute). Having said all that, an implementation of the Eclipse API that is backed by the VFS is certainly interesting. Although some performance and functionality would be lost, it would provide a way to quickly implement the API for file systems that VFS already supports. Similarly, KIO is an excellent abstract file system API for KDE, and a binding of our API to KIO would open up lots of doors very quickly.
(In reply to comment #14) > The main thing it lacks is support in the > API for progress reporting and cancelation. This should be easily possible to implement. > I also didn't see any way to carry over some important optimizations we have > made in Eclipse for file system access. You could replace the VFS local fileprovider with your native implementation. That way you should be able to loose not that much performance > These optimization > become even more significant in an environment where the file system is accessed > over a network. A network filesystem at all might not be that performant. However, where possible (e.g. ftp) VFS already tries to reduce network communication. Though, there is room for improvement. > Finally, the local file implementation of VFS doesn't seem to support setting > attributes such as read-only, executable, archive, hidden, etc, or even reading > some of those attributes (such as the executable attribute). That true, this needs native code and I tried to avoid it. But as I said, you can replace the local fileprovider. The next release after 1.0 will provide "services". On an fileObject you can query a "FilePermission" service and change its permissions then. For sure, you have to implement this "FilePermission" service and route it to your native library. > Although some performance and functionality would ... Yes, I have to admit I dont know how well VFS performs in an environment and use-case like eclipse. > Similarly, KIO is an excellent abstract file system But linux only, isnt it?
> - Clients calling IResource.getLocation().toFile() will continue to function as >always for the cases where the file is still in the local file system. In other >cases, they will get an exception. What kind of exception? I looked at the code in the branch and could not find a change here. Or did you mean clients will get an NPE when directly working with the result of toFile()? >In all >the cases I have found, there is an alternative to getLocation() that will work >for non-local files. Did you eventually document those cases together with the alternative?
> What kind of exception? IResource.getLocation() is already specified that it can return null, so currently I have it returning null in the case where the resource is not stored in the local file system. However, I am now leaning towards throwing an unchecked exception instead, which would allow me to give a clear error message that describes the situation. I think if getLocation() returned null in too many situations it would be very difficult to diagnose failures. > Did you eventually document those cases together with the alternative? Generally, getLocation() is called to obtain a java.io.File instance. Sometimes getLocation() can be avoided by calling API on IResource: IResource.getLocalTimeStamp() instead of File.getLastModified(), IResource.refreshLocal() followed by IResource.exists() instead of File.exists(), etc. In other cases, API on IFileStore can be used instead of methods on java.io.File. The only java.io.File methods that don't have an equivalent on IFileStore are createNewFile() and deleteOnExit() - these are somewhat special and aren't easily generalizable to other kinds of file systems. I will start working on a migration guide section for clients of IResource that want to support IResources that are not backed by the local file system.
For debugging it would be nice to be able to run with a special file system where every file access appears to be nonlocal, and getLocation() would always return null.
Re comment #18 - Good idea, will do. Thanks.
I looked at org.eclipse.core.filesystem (EFS) and at VFS. I must say, that I like the eclipse file system better, especially in the context of eclipse, because it supports IAdaptable, IPath and IProgressMonitor. I also like the idea of having progress monitors when fetching children. It's good that it is designed to allow more efficient native implementations than java.io.File. However, I have a few questions: - How do I get the full path of a IFileStore (I assume getName() returns the relative name to its parent) - How to get to an IPath from IFileStore? - I have the feeling that java.net.URI is pretty heavy weight (it's an object with 16 attributes) - How are links and symlinks handled? Eclipse currently has problems with recursive symbolic links (bug 105554)? There seems no attribute to specify if a IFileStore is a link. - How does EFS deal with changes in the file system. There seems no notification mechanism when files change. Does this mean, implementations have to poll for changes? That might be very expensive for remote systems. VFS supports a FileListener, which can be attached to a FileSystem. Listeners get notified on create, delete and modified events. - VFS supports arbitrary attributes (key/value pairs). How would that be supported by EFS (by IAdaptable)? There seems also a fundamental difference on what a file system is between EFS and VFS: In EFS a FileSystem represents a particular implementation (URI scheme). In VFS a FileSystem is the root of a tree -- a FileProvider is responsible for a specific URI scheme. I see several advantages of having FileSystem separate from the FileProvider: - FileSystems can be mounted/dismounted - The FileProvider is responsible to specify the capabilities of the FileSystem, therefore the same schema (e.g. ftp) could have different capabilities depending on the file system.
Michael, thank you for taking a look at this and for providing feedback. I should mention that this API is still in its infancy and thus we still have some freedom to change it, so this is a perfect time for people to be trying it out and giving concrete feedback like this. Here are some answers to your specific questions: Q: How do I get the full path of a IFileStore (I assume getName() returns the relative name to its parent) A: I wanted to abstract away as much as possible what the "path" of a file looks like, since this can be very different on different kinds of file systems. For example, a file system backed by CVS would need information like protocol, server name, repository path, file path, and branch name to be encoded in the file's path. URI was chosen instead of IPath because it provides this kind of flexibility. Q: How to get to an IPath from IFileStore? A: In general, you cannot, because of the reasons above. In some kinds of file systems it would be possible to use the path component of the URI (such as with a local file system): new Path(IFileStore.toURI().getPath()). My counter question is why would you want to perform this translation? Q: I have the feeling that java.net.URI is pretty heavy weight A: Absolutely, it's a big object. It also has almost no convenience methods that users of IPath are accustomed to: append, removeLastSegments, segmentCount, etc, etc. However, URIs are not necessarily held onto in memory - they are just used to construct an IFileStore instance. Also, you can often use API on IFileStore to avoid having to manipulate URIs directly. IFileStore getChild, getParent(), etc. In particular, IFileStore.getChild(IPath) can be used to obtain the child of a file system using a relative path. Q: How are links and symlinks handled? A: There is currently no specific support for symbolic links (as with java.io.File), although I will probably be added this. I didn't want to add too much change at once. It would simply be another attribute on the IFileInfo. Q: How does EFS deal with changes in the file system. There seems no notification mechanism when files change. Does this mean, implementations have to poll for changes? That might be very expensive for remote systems. VFS supports a FileListener, which can be attached to a FileSystem. Listeners get notified on create, delete and modified events. A: Currently there is support for this in the resources plugin (refresh monitors), but not in the filesystem plugin. It is true that VFS has API for this, although the only implementation I saw was a naive polling solution, which isn't much better than what clients can do. This is another thing that can easily be added to the filesystem plugin if the need arises. Q: VFS supports arbitrary attributes (key/value pairs). How would that be supported by EFS (by IAdaptable)? A: EFS supports arbitrary boolean attributes (IFileInfo.getAttribute(int)). If other non-boolean attributes were needed, new methods can be added to IFileInfo. IAdaptable could certainly be used to layer on other attributes that the IFileStore doesn't know about (such as the icon to use for a given file).
>I have it returning null in the case where the resource is not stored >in the local file system. However, I am now leaning towards throwing an >unchecked exception instead, which would allow me to give a clear error message >that describes the situation. While this is true it will bypass the client code that handles the null/exceptional case (hopefully) nicely. I'd vote against throwing an unchecked exception in this case.
(In reply to Johns comment #21) > A: I wanted to abstract away as much as possible what the "path" of a file looks > like, since this can be very different on different kinds of file systems. For > example, a file system backed by CVS would need information like protocol, > server name, repository path, file path, and branch name to be encoded in the > file's path. URI was chosen instead of IPath because it provides this kind of > flexibility. But now clients have to deal with both IPath and URI, because if you want to store somewhere the "path" of a file, you need an URI. How would be the relation between IPath and URI? How to store an EFS path? What would be the problem extending IPath with a protocol? (IPath in an interface that is not supposed to be implemented by clients, and therefore the platform can extend it.) How and when to use Path.fromOSString Path.fromPortableString, IPath.toPortableString, when to use the URI? VFS has FileName, which is a mixture of IPath and URI... > Q: How to get to an IPath from IFileStore? > A: In general, you cannot, because of the reasons above. In some kinds of file > systems it would be possible to use the path component of the URI (such as with > a local file system): new Path(IFileStore.toURI().getPath()). My counter > question is why would you want to perform this translation? Ok, what is then the relationship between IPath and IFileStore? Maybe I'm a little bit confused what EFS is supposed to be used for. I thought is a kind of general file system abstraction (and java.io.File replacement) in the context of eclipse. I also thought IPath is a kind of eclipse URI.
My thinking is that you only need URI to represent an IFileStore, such as for serialization. IFileStore.toURI(), and FileSystemCore.getStore(URI) are the symmetric operations for translating between the two. IPath is only used in this API as a convenience: IFileStore.getChild(new Path("a/b/c")) is the same as IFileStore.getChild("a").getChild("b").getChild("c"). So, I guess the reasoning is that URI is too powerful, and IPath is not powerful enough, so we should invent something new. I think the end result would be very similar to URI: We need something to identify the "scheme" to figure out what kind of file system it is, and what extension will provide an implementation for it; the "scheme-specific" part should be as flexible as possible to accomodate all the different kinds of file systems that might want to be plugged in; it may need a host and port, it may need user info, etc. Given that we need something very URI-like, and URI is a very well known and widely accepted standard, it seems like unneccesary effort (and learning curve for clients) to create something new or to augment IPath with these extra features.
John, you see no problem that eclipse will use URIs and IPaths in paralell? Won't there be cases where IPaths don't work and URI has to be used instead (like storing a history in some dialog settings etc)? I'm just asking (I really don't know the answer). To me, it just feels like trouble if there are suddenly 2 ways to represent a file in eclipse path: IPath and URI.
Couple questions: In the case of non-local resources, IResource.getLocation() will return null, correct? I think that IPath may prove inadequate for database-oriented local file systems anyway (I'm thinking about WinFS). The way to specify the name of a file will not be a simple list of segments separated by slashes anymore. Whether you use a URI or something like VFS's FileName, I believe that many uses of IPath in Eclipse are going to have to be rethought, not just ones touching IResource. Regarding notifications, even if current VFS implementations use polling, the implementation is hidden. If the file system provider supports asynchronous notifications then the client could get a boost without having to change a line of code. There should probably be a way to ask if async notifications are supported so the client can decide whether or not to use notifications at all. For example, if async notifications are not supported I might decide to add a 'refresh' button that the user has to click on instead.
> If the file system provider supports asynchronous > notifications then the client could get a boost without having to change a line > of code. I already started a separate project at http://l3x.net/imwiki/Wiki.jsp?page=JFileWatch which currently supports inotify (linux) to allow file system notifications in java. Though, I dont know if it works with the latest inotify version, and windows support is still missing. But if there is a someone else who might provide code for windows (and maybe mac) we could try to integrate it in VFS.
Some answers (note I have started to use "EFS" to refer to this Eclipse file system API to distinguish it from other file system APIs): Q: Will IResource.getLocation() return null if the resource is not in the local file system? A: I'm still undecided on this. It would be reasonable according to the existing spec of getLocation() to return null in this case. The spec says it returns null "if no path can be determined". The other option would be to have getLocation() throw an unchecked exception in this case. Throwing an exception makes error reporting clearer, and better than the NullPointerException most existing clients get (those who call IResource.getLocation().toFile()). I am also considering deprecating the method. Comments welcome on what the best approach would be. Q: Wouldn't it be better for EFS to support notification? A: Yes, ultimately I think it would make sense for file system notification to happen at the EFS level. It's not very compelling at the moment because we only have native notification support for windows, but if inotify becomes broadly available the argument for adding such API is much better (anybody know of an inotify for OSX?). Note we have an existing enhancement request for inotify (bug 108697) - after spending weeks trying to get famd working on Linux I can't spend too much time on it right now, but it would be a great way for someone to make an Eclipse contribution (hint, hint ;)). I'm still pondering the IPath/URI thing... I'll follow up on that later.
I don't like the unchecked exception thing. It's spec'd to return null, so if it returns null it shouldn't be an exception. Throwing an exception will be an API change. See also: http://www.eclipsezone.com/eclipse/forums/t49685.html
In reply to comment #21) > A: Currently there is support for this in the resources > plugin (refresh monitors), but not in the filesystem > plugin. It is true that VFS has API for this, although > the only implementation I saw was a naive polling > solution, which isn't much better than what clients can > do. This is another thing that can easily be added to the > filesystem plugin if the need arises. I think polling would not work in some cases. Suppose, I want to create an EFS that connects to a remote eclipse. This file system would represent the IResources in the remote eclipse. Polling would not help refreshing the remote system. And if the remote system changes, the local system would not know about the changes. Therefore, I think the API needs to support refreshing and change notification. BTW, I think the remote eclipse file system would be a great example to prove that EFS is sufficient to support the eclipse resource model.....
How does EFS relate to http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout% 7E/platform-core-home/docs/eclipse-remote-dev-guidelines/remote- development.html ?
Here is a way of thinking about the division between IPath and URI. A URI allows you to uniquely identify a particular file system in the universe of possible file systems. An IPath allows you to locate a single file or directory within a *particular* hierarchical file system. Essentially, IPath is always relative to some known root node, but it can't help you locate or identify a root node in an API where more than one is available. IPath was sufficient in the past because there was only one file system we ever cared about: the local file system as represented by java.io.File. We now allow arbitrary file systems to be installed, so something like URI is needed. However, this does not mean that all client code needs to deal with URIs. Once you have an IFileSystem in hand, you can lookup files and directories using IFileSystem.getStore(IPath), much like you could in the past. In other news, I have started a document that describes how clients can transition towards supporting these more flexible workspaces (note our URLs no long uses the clunky "viewcvs" utility - yay!) http://eclipse.org/eclipse/platform-core/documents/3.2/plan_flexible_workspaces.html
Re comment #31: First, here is a nifty new URL for the same document: http://eclipse.org/eclipse/platform-core/documents/2.0/remote-development.html Although this is an old document, it is still very interesting reading for anyone considering different development styles. The different development modes that it describes (workspace vs. remote resident and shared vs. replicated) are all still valid. The EFS support essentially enables the "shared access to remote system-resident file base" mode. This mode still has the drawbacks of not easily supporting offline development, and only being viable if file system accesses are infrequent or the remote file system is very fast. I think the general recommendation in that document still holds: that development based on local replication is best in most cases. However, there are situations where development on non-relicated remote systems is desired and indeed much simpler, such as when clients are always connected to the remote file system over a fast and reliable connection. Those are the kind of scenarios we are trying to support with this new API.
I've only glanced at new API, but ... seems some obviously missing methods are getCharset and getContentType (or ... for consistency, maybe getContentTypeDescription). Seems these are essential to handle "streams" correctly. And ... before you even suggest it :) I'd prefer these be "built in" methods on IFileStore, rather than "hidden" attributes of IFileInfo since, to me, these are fundamental characteristics of "files stores" ... nearly as important as the contents themselves :) And, since some sheme's (e.g .http) have this info "built in" to their specs, and some do not, it might be good to "plan ahead" and have something like a "supportsProperty(String)" method on IFileStore or IFileSystem. I imagine there would be other "properties" that may or may not be supported by particular types of file stores, so thought I'd mention my perceived need of a "generic" way of determining. (Or, is this intended to be provided by IAdapter?). This way, for example, if a scheme's protocol was not spec'd to provide "charset", then the "normal eclipse rules" could be used. cool stuff.
Perhaps, I'm misguided, but this issue deals strictly with the logical to physical separation of the eclipse resource API's correct? In that, it doesn't address the need/ability to have multiple projects that are physically nested, but logically they could be represented in eclipse as sibling projects. For example On Disk: Maven 2.0 Project A/ Maven 2.0 Project B/ Maven 2.0 Project C/ In Eclipse: Maven 2.0 Project A/ Maven 2.0 Project B/ Maven 2.0 Project C/
Correct. This bug does not deal with nesting or hierarchical organization of projects either in the workspace or on disk. See bug 35973 for that request.
I am new to Eclipse development, so forgive me if I make a stupid comment out of ignorance :), but I was very excited to find this enhancement request. I will be reading the org.eclipse.core.filesystem API in depth and will be providing more comments soon. I work for a national lab, and we will be extending Eclipse as an application framework for a different domain (biology). Due to security constraints, our users will not be able to store their project data on the local filesystem, so having an extension point for file storage is essential. We plan to connect to a remote server via WebDAV. Our security requirements dictate that all information be stored remotely, including metadata, so we would like to see marker storage extensible as well. We also plan to pick up change notifications via JMS, so we would like to extend the observance mechanism too. Would comments on these two topics go here, or is there a more appropriate bug to post to? We hope to work closely with you so our changes follow the Eclipse migration path. We are also willing to contribute our work back to Eclipse, if it is helpful.
(In reply to comment #36) > Correct. This bug does not deal with nesting or hierarchical organization of > projects either in the workspace or on disk. See bug 35973 for that request. Hmm, that puzzles me. I thought that it would be possible to implement a file system, that maps an arbitrary directory structure to a set of projects. That would allow, to keep e.g. the .project file outside the project tree. It would also allow to have project1 an eclipse project as well as project2 and project3: project1/ project2/ dir/ project3/ otherfies/ file1 file2 The file system could also deal with symbolic links: project/ dir1/ ...many files dir2 -> dir1 dir3 -> dir1 This causes currently 3 times the number of files in a project, just because of symbolic links. An implementation of a file system could break those links. I don't see how bug 35973 addresses those issues, because working sets are not the solution to those problems. bug 35973 comment 9 even says: "...no current work is planned in this area.." and it is kind of marked complete.
I have been adding comments to bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=84988 In regards to mapping projects to nested folder structures (such as Maven) using filters. Your feedback is welcome here...
Re comment #38 - sorry, I mixed up hierarchical projects in the workspace with hierarchical layout on disk. Since EFS provides a complete level of indirection between the workspace and the file system, it is true that an EFS file system provider could cause sibling projects in the workspace to come from overlapping locations on disk. With resource exclusion filters (bug 84988), you could ensure the overlapping portion on disk did not cause duplication within the workspace. What this bug does not deal with is nesting of projects in the workspace (an IProject containing another IProject). That's what bug 35973 is about, and it's accurate to say no work is planned in that area. Sorry for the confusion.
Re comment #37: Thanks for your interest and your offer of support. You should be aware that were are not currently considering redirecting file system access for arbitrary Eclipse metadata (things in the workspace/.metadata directory). This would be *much* more ambitious than our current scope. In case you aren't aware of it, Eclipse does have a non-versioning WebDAV implementation, although it is not currently bundled with the Eclipse SDK (org.eclipse.webdav plugin, available from the downloads page). General comments on this plan item and on the file system API are welcome in this bug report. If you have additional feature requests, or have discovered bugs, then a separate bug report would be more appropriate.
John, it's good to see that supporting nested file system layouts would be possible. What would be even better is if eclipse then came with this support. I think the # of developers who could benefit from such an EFS implementation is greater than developers trying to use FTP as a remote file system.
I have created a central place for documents relating to this plan item: http://eclipse.org/eclipse/platform-core/documents/3.2/flexible_workspaces/ On this page I have added a document that tries to capture all the ideas and requests raised here, linking them to progress information and corresponding bug reports.
First off, I'm really excited about the direction this item seems to be taking. Also, w.r.t symbolic links and equivalent constructs, I'd be happy with most any EFS that parallels SWT's widget tack. I know adopting a particular file system would be ruinous, but how about composing API on a native fs (or two) that does support the behaviors we'd like to see. For your consideration, along those lines: "Plan 9 has no symbolic links, but its name space construction operators, bind and mount, make it possible to build the same sort of non-hierarchical structures created by symbolically linking directories on Unix." http://www.cs.bell-labs.com/sys/doc/lexnames.html Also, for Linux/Unix users: http://v9fs.sourceforge.net/ And the excruciatingly absent attachment is: Reference win32 9p client (v9fs via ssh), abstracted through EFS. Perhaps via Tim Newsham's (wonderful) Python 9p implementation? http://www.lava.net/~newsham/plan9/ Since the phrase 'widget tack' isn't very helpful, here's a pseudo-quote from the Eclipse guide, with a couple of name replacements: "[EFS should] define a common portable API that is provided on all supported platforms, and implements the API on each platform using native [file systems] wherever possible. This allows the toolkit to immediately reflect any changes in the underlying OS ... while maintaining a consistent programming model on all platforms."
Ryan, your comparison with SWT is a good one. The aim of the EFS is to provide a simple API for accessing native file systems, including file systems accessed over a network. As with SWT, the main client is other plugins in the Eclipse platform, but the API tries to be general enough that it can be of use to others. Unlike SWT, an additional goal of EFS is to impose no porting overhead on Eclipse. Where we have native code, it provides either optimizations on things provided by java.io, or optional extras that clients can live without. If you are running on a platform for which we have no natives available, clients of EFS should be affected. I'm a bit fuzzy on where you were going with your references to 9p. Are you suggesting the EFS API be modeled on the 9p protocol, or simply that 9p would make an interesting reference implementation?
(In reply to comment #45) > ... an additional goal of EFS is to impose no porting overhead on Eclipse Excellent, and well said. > I'm a bit fuzzy on where you were going with your references to 9p. Are you > suggesting the EFS API be modeled on the 9p protocol, or simply that 9p would > make an interesting reference implementation? Yes. I'd say that 9p could make for an interesting implementation that would be suitable for comparison to ftp, and would certainly contribute additional usage scenarios to the EFS model. For instance, I've going started investigating name space binding and synthetic recourses in Plan 9 to inform my notions regarding symbolic linking and virtual nodes within the workspace. I should also reiterate that I like the current API and that I'm principally interested in operation semantics, not names.
Correction to coment #45 - this sentence was missing a very important "not": If you are running on a platform for which we have no natives available, clients of EFS should NOT be affected.
Is this planned for a particular milestone?
> Is this planned for a particular milestone? This work is being rolled out over several milestones. The following was in 3.2 M3: - Eclipse file system API (org.eclipse.core.filesystem), aka "EFS". - Resources plugin rewritten on top of EFS instead of java.io.File - API to create a project given an arbitrary URI (see IProjectDescription.setLocationURI) - Support in the text editor framework for opening editors on files in arbitrary file systems. - Initial work in JDT to make it run on arbitrary file systems (ongoing) And in this week's build: - API to create linked resources to arbitrary URIs. The core API work on this plan item is nearly complete. There is currently no UI for creating resources at arbitrary locations, or for browsing other file systems. And I'll take this opportunity to ask those who are planning or considering implementing the file system API to look at this API soon and start trying it out. The window during which API changes can be made for 3.2 is closing fast.
I have been looking at this API and have a couple of questions - 1) Is it going to be possible to create a project on the local filesystem containing IFiles on a different file system? 2) Is that what linked resources based on URIs supposed to allow? If thats the case,are all the plugins (like CVS that do not allow actions on linked resources) going to support IFile actions on linked resources as well? It seems from comment 49 that the only thing that decides filesystem is the setLocationURI on IProjectDescription. Then resources under IProject can use IPaths to indicate their relative locations under the project. Is this correct?
Q & A from comment #50: Q: Is it going to be possible to create a project on the local filesystem containing IFiles on a different file system? A: Yes. Q: Is that what linked resources based on URIs supposed to allow? A: Yes. Q: If thats the case, are all the plugins (like CVS that do not allow actions on linked resources) going to support IFile actions on linked resources as well? A: No. Some plugins have a strict dependency on the physical layout of files in the local file system. This includes all external tools such as Ant, Maven, etc, and most repository systems. Any plugin with such a strict dependency likely does not work with linked resources today, and likely won't do so in the future. Such tools will also typically ignore resources stored in other kinds of file systems. In the case of repository tooling, the assumption is that the repository tooling knows how to interact with some particular file system (whether local or not). Q: It seems from comment 49 that the only thing that decides filesystem is the setLocationURI on IProjectDescription. Then resources under IProject can use IPaths to indicate their relative locations under the project. Is this correct? A: Yes. IProjectDescription location and linked resource location are the two points where a particular file system can be associated with a resource. For resources below a particular project or linked resource, they can be dealt with in terms of their IPath relative to the project or linked resource. I.e., each resource does not know its file system location directly, but each resource knows its parent, and one of its ancestor resources will know its file system location.
Another question about URI: Is it a concern that java.net.URI was added in 1.4 (it has @since 1.4 tag in javadoc)? I mean will that limit where this code can run in any significant way? I notice that platform code and OSGi code uses URLs instead of URIs (e.g. Platform.resolve(URL). I'm not sure if this is historical or intentional (to allow platform to run on as many versions of Java as possible). I see that org.eclipse.core.resources uses URI also but all of the API usage looks like @since 3.2 (and the internal usage mostly seems to be supporting the 3.2 work).
It is unfortunate that using URI forces org.eclipse.core.filesystem to use Java 1.4 or greater, but it is not uncommon for Eclipse plugins to require 1.4 (see the Eclipse 3.2 plan, which lists the minimal execution environment for each plugin). The org.eclipse.core.resources plugin has depended on 1.4 since the Eclipse 3.0 release. The runtime and OSGi plugins, on the other hand, have a firm requirement to run on a minimal class library (J2EE Foundation 1.0, or even less for OSGi). URI and URL are not interchangeable: URL has more restrictive syntax, and more importantly requires a URLStreamHandler to be installed with the JRE which performs validation of the syntax (i.e., URL is more expensive to create). The class comment of java.net.URI has a nice summary section called "URIs, URLs, and URNs", that does a good job of describing the conceptual differences between URI and URL (to save the slog through RFCs).
First, forgive my ignorance. I have been reading through this and also some related bugs (which actually is a lot!), but I am still not sure I understand everything. I have a few rather general questions regarding build systems using external builders and build tools (where e.g. make, and subsequently all compilers, are called outside the Eclipse context, knowing only about the physical file system), and how they are considered to interoperate with such logical file system implementations: How should a build system based on external tools deal with file systems not providing physical path information for parts or all of its content (and where no path mapping is possible either - I understand this could happen)? Is there general information of a logical file system implementation available indicating to the build system, it cannot rely on getting an IPath or IFile object for none or at least not all entities of this particular file system? How do you envision this issue to be addressed by any existing build system implementation using external tools (because, of course, the physical file system has no chance to obtain file system information in such cases)?
In general, interop with external tools and support for a file system abstraction are mutually exclusive goals. If you abstract away from the local file system, you won't get free interop with external tools that operate only on the local file system. Interestingly, a major motivation for abstracting access to the local file system is to gain the ability to interoperate with external tools that operate on *other* file systems. I.e., an IResource can only be associated with one file system at any given time. This allows interoperability with any external tools that operate on that particular file system. The choice of file system will typically be determined by the set of external tools the user wants to employ. For example, if you are running with build technology that operates against a remote file system (a typical setup for embedded or server-side development), this plan item allows you to create IResources stored on that same file system. Of course, you can't do this while simultaneously expecting external tools operating in the local file system to continue working. The question of how external tools are expected to handle disparate file systems is good. My expectation is that the Eclipse action responsible for invoking that tool with ask the IResource for its location, and only call the external tool if that location is appropriate for that tool. The local file system case is easier because you can use the old IResource.getLocation() method: IPath location = IResource.getLocation(); if (location != null) runExternalTool(location.toFile()); else //don't run the tool, optionally show an error dialog Ideally this could be done during action enablement calculation to prevent the action from being available to begin with in cases where it is not appropriate. Of course, a richer integration could be enabled by creating a copy of the file in the appropriate file system before running the external tool. I don't think this is necessary in the general case, but might be appropriate in some situations.
First I think this is a great idea. I would like to add that returning null in cases where things aren't handled, is at best a vague solution. I agree that some sort of context or status as to why things failed would be nice. Perhaps some sort of status can always be retreived to find out what happened, such as GetLastError() or some similar thing. Another way would be to throw some form of exception.
Re comment #56. I experimented with throwing an exception from getLocation() rather than returning null. However, it turned out that this caused code that previously handled null locations to start failing. I.e., when returning null, most of the SDK "just worked" without further changes required. When having it throw an unchecked exception, it caused failures all over the place. I agree an exception would be much more descriptive, but it seems to do more harm than good in practice.
Work on this plan item is complete for the 3.2 release.
