Community
Participate
Working Groups
I am working on a utility class that allows for path manipulations on EFS resources. The URI format that a given file system uses though might not use the path field of the URI (or might use it for something, but it might not contain the physical path of the file as seen by the target system). Parent child relationships in EFS might be virtual as well, and so one cannot just assume that by concatenating all the names of an IFileStore's parents together with path separators in between them, that one can construct a valid path. As such, it's become impossible for me to create a version of my class that works generically for any arbitrary file system. I would like to see an IFileStore.getPhysicalLocation() method or the like (name it what you want), that returns the physical location of the file as it would appear on the target system. This way the provider of the file system can provide the implementation for this. This information will be required regardless for interfacing with external tools such as make utilities, compilers, and debuggers. External tools know nothing about EFS or Eclipse resources, so we have to be able to know what the actual path to the referenced file is.
Depending on what you mean by "physical file", I think we already have this... IFileStore#toLocalFile. EFS implementations can override this method and implement it however they want. Note that I believe the resources plugin does assume that the path portion of an IFileStore's URI can be manipulated in order to get at the parent/children. An IFileStore that does not maintain this relation will likely break in strange ways when IResources are created backed by such file stores.
(In reply to comment #1) > Depending on what you mean by "physical file", I think we already have this... > IFileStore#toLocalFile. EFS implementations can override this method and > implement it however they want. But... if it's not a local file, so we can't return java.io.File. We need something that works for any type of file. Doug, if the resources plugin is making these assumptions, maybe your Flexible File System will have to work a bit differently.
(In reply to comment #1) > Note that I believe the resources plugin does assume that the path portion of > an IFileStore's URI can be manipulated in order to get at the parent/children. > An IFileStore that does not maintain this relation will likely break in strange > ways when IResources are created backed by such file stores. Now the URI path does give parent/child as far as IResource is concerned. But you can't assume the java.io.File parent/child will match. Is that OK, John? And actually I'm currently thinking that we'll reuse linked resources for the Add so things won't be so different for IResource. We'll just be hiding children from the IResource system.
> But... if it's not a local file, so we can't return java.io.File. We need > something that works for any type of file. What representation would this physical location have? URI is the only representation I can think of that can represent the physical location of a file in an arbitrary file system.
> But you can't assume the java.io.File parent/child will match. Is that OK, John? Yes, that's fine.
(In reply to comment #4) > What representation would this physical location have? URI is the only > representation I can think of that can represent the physical location of a file > in an arbitrary file system. Hmmm... well I guess you have a point there. But the fact remains that in our case we are assuming that somewhere, eventually, the file ends up mapping to a physical file in some operating system, which has some sort of path associated with it. Tools like CDT just plain won't work on files hidden in databases or what have you, because the external tools most likely won't know how to get at the files.
Chris, I've been thinking about this issue and I still need more details. The best would be if you provide us some steps illustrating what you are trying to do. If I understand you correctly, the best solution would be to mix a custom EFS implementation and linked resources. However I will wait for more details.
Thinking about this a little more, I guess one of the potential uses of EFS is to store files in repositories that don't necessarily have physical file locations. So I can see the resistance to returning information like that. I guess for our needs, we may need to be more explicit about what file systems we support. One idea I mentioned at the CDT summit was to have the file store implement a separate interface to return such information. You could then get at that information using an adapter. If the getAdapter returns null, there is no physical location. If it returns an object, you can use it to get the location. That would obviously require a change to LocalFileStore.
(In reply to comment #8) > I guess for our needs, we may need to be more explicit about what file systems > we support. One idea I mentioned at the CDT summit was to have the file store > implement a separate interface to return such information. You could then get > at that information using an adapter. If the getAdapter returns null, there is > no physical location. If it returns an object, you can use it to get the > location. That would obviously require a change to LocalFileStore. > I assume that you want to use the fact that IFileStore extends IAdaptable and use the org.eclipse.core.runtime.adapters ext. point to register an adapter's factory? What kind of change to FileStore are you talking about?
(In reply to comment #9) > I assume that you want to use the fact that IFileStore extends IAdaptable and > use > the org.eclipse.core.runtime.adapters ext. point to register an adapter's > factory? > > What kind of change to FileStore are you talking about? Oh wait. You're right. I guess I didn't think it all the way through. Certainly we can register our own adapter and have it do anything special we need.
(In reply to comment #7) > Chris, > > I've been thinking about this issue and I still need more details. > The best would be if you provide us some steps illustrating what you are trying > to do. The scenario is the following: We are trying to support CDT projects being hosted via arbitrary EFS providers. C/C++ compilers (and other languages too) are typically external processes launched by Eclipse. Hence, when interacting with those tools (i.e. the user builds their project), we can only reference files in terms of paths in the operating system, and not EFS, as the compilers are not an integral part of the IDE. There is no universal way to get at the underlying path to an EFS resource. An EFS filesystem provider can choose an arbitrary structure for the format of the URIs that their filesystem consumes. The path field of the URI might for example only reference the path to the resource in the workspace and might not have any connection to the location of the file in the operating system. Hence, without knowing the structure and interpretation of the URI for a given filesystem, it is impossible to know for sure how to reference that file outside of Eclipse. Furthermore, some files may not have a physical representation in the operating system, and hence we also need to know whether or not a file has a physical representation.
(In reply to comment #11) Chris am I getting this right: Assuming a remote workspace on a VMS system, you want to compile C files on that VMS system. In VMS, the root folder is [PARENT.DIR.PROJECT] and the Makefile is [PARENT.DIR.PROJECT]MAKEFILE;10 you have an EFS Provider which maps this to URI ssh://myvms.foobar.org/parent/dir/project/Makefile so what you need is either String IFileStore#getNativeAbsolutePath() returning what the path looks like in VMS, such that you can ssh or telnet into your remote VMS shell and "cd" to the appropriate directory; or, you register an class VMSNativePathAdapter { String getNativeAbsolutePath(IAdaptable fileStore); } that would be capable of translating the URI into what VMS wants. I think it's true that the "native absolute path" is something that might be interesting for generic remote file systems, when it comes to interfacing the file with a shell service on the remote system. For some file systems (in databases, for instance), a unique record ID could be the "native absolute path". On yet other systems, the query could be meaningless. Would you need the nativeAbsolutePath to be canonical in the sense that multiple paths (e.g. via symbolic links) to a given item all yield exactly the same canonical remote path? If you think you need this, then I guess implementing it would be harder, and it would potentially be slower. In terms of backward compatibility, when IFileStore#getNativeAbsolutePath() does not exist or return null, a client could fallback to taking the "path" portion of the URI. But I'm just guessing. Do you have a concrete example where you cannot just take the "path" portion of the URI, and let us know why you cannot take it?
(In reply to comment #12) > > But I'm just guessing. Do you have a concrete example where you cannot just > take the "path" portion of the URI, and let us know why you cannot take it? > Isn't your example exactly that? Give the path portion of the URI "parent/dir/project/Makefile", how would it be possible to translate this to [PARENT.DIR.PROJECT]MAKEFILE;10 without the EFS implementation providing the mapping?
Well... you are right of course. What I meant was that in the VMS case of my example, the mapping of "native" paths to "URI" paths is uniform and thus relatively simple: String[] components = path.split("/"); StringBuffer buf = new StringBuffer(); buf.append('['); for(int i=0; i<components.length-2; i++) { buf.append(components[i]; buf.append('.'); } if(components.length>1) { buf.append(components.length-2); } buf.append(']'); buf.append(components.length-1); return buf.toString(); so the transformation can also be done separately from EFS (in an adapter, for instance, or in the shell subsystem). This is not possible if the transofrmation needs to be non-uniform in any way such that knowledge about a node is required in order to transform the path to that node. And then, I think that on VMS there is also a "UNIX-like-subsystem" that on could use for compiling, instead of using the native file system. Not sure if such a "UNIX emulation" exists on other hosts. MVS aka z/OS has it as far as I know, I'm not sure about others.
I have worked up two draft solutions to this issue (and several others pertaining to the logical vs. physical issue). Both draft solutions are API compatible with 3.5. Both probably require some cleanup before they would be comitted (e.g. cleanup @since tags). The patches are currently against the e4 repository but could be trivially reworked to go into 3.5. Approach 1: =========== This approach declares a new interface, IFileStore2, which provides new APIs which specifically provide either "logical" operations (i.e. as the workspace sees the resources) or "physical" operations (i.e. as the operating system sees the resources). Previous methods on IFileStore which were ambiguous (and thus, were defacto logical operations) are deprecated. Pros: Ambiguous APIs are explicitly discouraged. Easy for API consumers to figure out what they should be using instead (e.g. all implementations of IFileStore in the platform would now implement IFileStore2, which readily shows up in things such as Type Hierarchy). Cons: More code changes were required in order to provide new functionality in a non-breaking way. Approach 2: =========== This approach declares a new interface, IPhysicalStore, which encapsulates the physical operations from Approach 1 into their own interface. IFileStore is unchanged, and logical operations are not explicitly defined (logical operations remain the purview of the ambiguous APIs in IFileStore). Implementations of IFileStore in the platform become adaptable to IPhysicalStore. Pros: Less code changes required. Cons: Ambiguous APIs remain. Difficult for API consumers to know that they should be adapting to IPhysicalStore where applicable (consumers are unlikely to notice usage of IPhysicalStore unless they happen to inspect the code in GetAdapter()). There is a possible third approach, if this work were solely going to appear in e4, namely, that the new APIs could be added directly to IFileStore, and the ambiguous APIs removed altogether, but as that would be a breaking change I have not gone down that path, as I am hoping this could make it into 3.5. Patches to be attached shortly. Feedback is appreciated.
Created attachment 121665 [details] Draft patch for approach 1. Derived from e4 repository. Draft patch for approach 1. Derived from e4 repository.
Created attachment 121666 [details] Draft patch implementing Approach 2. Derived from e4 repository. Draft patch implementing Approach 2. Derived from e4 repository.
Looking at the patches, I find both of the proposed approaches very confusing. The original request (as I understood it) was to add a single method to get the physical file system path, but the patch here has a whole raft of new methods: getPhysicalChild, getLogicalChild, getPhysicalParent, getLogicalParent, etc. I don't see how clients will be able to make sense of any of this... I certainly couldn't. I think of IFileStore like this: IFileStore presents the illusion or abstraction of a simple tree structure. Regardless of what the physical backing representation is, it is presented to clients as a tree. Trees have simple rules that any developer will understand: each node has a parent (except the root), and can have any number of named children. The IFileStore getParent and getChild methods allow a client to navigate through this tree in a predictable way. Since the resource API expects the underlying file system to be a tree, this allows an EFS to decide how they want to map their arbitrary backing store onto a tree structure. In the proposed patches, each is now situated in a two dimensional tree, with two parents and two disparate sets of children. Presumably if you call getPhysicalChild("x").getLogicalParent(), you will arrive somewhere different from where you started. I find this very confusing to think about, and wouldn't want to impose this complexity on clients lightly. The example Chris gave for this in the e4 call was a file system containing a zip file. A zip file can be "expanded" as a sub-tree of the file system so that entries in the zip appear as file system elements, as some operating systems do with their file system explorer views. To me this "expanded" tree still has traditional tree characteristics - each node with a single parent, and a set of children described by a simple name. I really don't see what the methods getLogicalParent versus getPhysicalParent will do in a case like this. Chris, maybe you can give examples of when any of these methods are needed and what kinds of clients would use getPhysical* versus getLogical*. Which methods would the IResource implementation use?
The concept of logical vs. physical paths is also important when identifying aliases of a file system path, which are induced by hard links or symbolic links (as requested by bug 233939). In this case, multiple logical paths (aliases) can be viewed as referring to a single physical path (the "canonical path"). I tend to agree with John that the concept of a "physical parent" vs. "logical parent" is confusing or probably sometimes not even well-defined. Bug 253705 may also related in terms of aliasing. Another scenario is a case where an IFileStore is wrapped inside multiple providers: Imagine a case where a HTTP provider gets a TAR file, part of which is a ZIP file, from which a TXT file is extracted: http://myhost/myfolder/myfile.tar subpath/subfile.zip zippedPath/myfile.txt I could certainly think about a system that's capable of extracting the TAR / ZIP files on the fly such that myfile.txt can be edited. Perhaps what Chris really thought about was the notion of a "file system container"? Then a method URI IFileSystem#getContainer() might be in order, such that the "physical parent" of an IFileStore is IFileStore.getFileSystem().getContainer()
(In reply to comment #18) > Looking at the patches, I find both of the proposed approaches very confusing. > The original request (as I understood it) was to add a single method to get the > physical file system path, but the patch here has a whole raft of new methods: > getPhysicalChild, getLogicalChild, getPhysicalParent, getLogicalParent, etc. I > don't see how clients will be able to make sense of any of this... I certainly > couldn't. Well I made a point to put in plenty of javadoc but perhaps it was not enough... > The example Chris gave for this in the e4 call was a file system containing a > zip file. A zip file can be "expanded" as a sub-tree of the file system so that > entries in the zip appear as file system elements, as some operating systems do > with their file system explorer views. To me this "expanded" tree still has > traditional tree characteristics - each node with a single parent, and a set of > children described by a simple name. I really don't see what the methods > getLogicalParent versus getPhysicalParent will do in a case like this. Well from the javadoc, something like a file that lives inside an archive file is not considered a physical file, because the OS does not view it as a distinct file. Hence all those "physical" methods would throw exceptions because you're trying to do physical operations on a non-physical file. The logical operations are the only ones that apply to the zipfile case, because that filesystem is inherently a logical one vs. a physical one. >Chris, > maybe you can give examples of when any of these methods are needed and what > kinds of clients would use getPhysical* versus getLogical*. getLogical* is used by anything that only cares about resources as Eclipse sees them, which is not necessarily the same as what is on disk. For example, say we have a file foo.c, which is mapped into the resource tree as bar.blah CDT operates with external tools. E.g., we call out to an external compiler in order to do a build. Calling the external C compiler on bar.blah is useless, because that file does not exist on disk. The compiler will fail with an error because it will look for bar.blah and find the file does not exist. We need to tell the compiler to look for foo.c instead because that is the file that actually exists on disk, and the compiler, not being hosted in Eclipse, knows nothing about EFS. As another example, CDT has its own parser that parses C/C++ files in order to do things such as search, content assist, etc. In order to parse properly, this parser must behave in the same manner as the external compiler will when we do a compile. So, when its built in preprocessor sees an include statement such as #include "../foo.h", it needs to take that relative path and find it on the target system. However, the parent reference needs to be resolved against the physical file tree as the disk sees it, and not against the logical tree as Eclipse sees it, because again, the compiler has no knowledge of Eclipse's view. So, if I have an EFS provider where all the folders are virtual and the resources contained in them are essentially linked in (but not linked resources in the conventional manner), following the parent reference via the current getParent() will return the wrong result. The ../ will resolve to the parent virtual folder, and not to the physical parent. Then probably I try to get a foo.h child from that virtual folder, and there probably isn't one, so my preprocessor fails because it can't find the file. I see now there is also another case which my proposals didn't handle. It's not really directly related to the logical vs. physical debate, but it's important, so allow me to digress for a moment... Say we have another include statement that says #include </usr/include/foo.h> Now say for example that we have a weird EFS provider that has URIs of the form myprovider://myserver.ibm.com/database.cgi?/usr/include/foo.h The path component of the above URI is actually database.cgi, and is never used to hold the path of the actual file/directory in question. Rather, the query portion is what you actually need to work with paths as the OS sees them. As a tools provider that wants to be able to support any arbitrary EFS provider without special knowledge about them, this makes things difficult. Without knowing the implementation of the filesystem and hence also not knowing how this URI is structured, how do I create a URI for that filesystem that goes to an arbitrary, absolute location in the filesystem? The location has never been seen before by EFS as I have not browsed to it or anything like that, all I know is I have this absolute path and I have to do something with it. In IFileSystem, we have: public IFileStore getStore(IPath path); which says it's just a convenience method that makes assumptions about using the path component of the URI. So, as an EFS consumer, I have no real way of handling that situation with the current API. >Which methods would > the IResource implementation use? The IResource tree is the logical one, so it would use the getLogical* methods.
I'm curious to hear what the other e4 resources folks think... to me it feels strange to be pushing the logical IResource structure down to the EFS level. This could be supported by having separate links at the IResource level for each place where the resource tree structure diverges from the physical file system structure. This would allow EFS to map only to the physical structure. I can understand the use case of a zip/tar being expanded in place to become part of the file tree, but that by itself doesn't require introducing logical/physical parent/child concepts. I think all we would need for that particular case is the API described in comment #0 - a method to get the physical path of an IFileStore. For the #include "../foo.h" case I also think you just need the "getPhysicalPath" method. To resolve the include you'd do store.getPhysicalPath().append("../foo.h"), or something similar depending on the return type of getPhysicalPath().
I haven't thought about this issue for a while so excuse me while I try to do a reset and get back to where we are. The purpose of this feature is to allow tooling to interact with external tools that need to know the location of resources using paths they understand. In particular, we need to be able to generate a string for those paths so we can send it on the command line. Also, we need to be able to translate paths back into resources for things like compiler error parsing which creates markers. Now for the vast majority of systems we work with using Eclipse, the toLocalFile() method on IFileStore works. But there are cases where the files are remote and can't map to a File. And I'll somewhat ignore the case where the file is in some other repository or zip file or what have you since that rarely exists in the field if at all. What I think we need is really a remote capable version of java.io.File. We could write a Local implementation that delegates to File, and we can provide various remote implementations that use things like ChannelSftp from jsch. We could then provide a toPhysicalFile() or an adapter (which is probably what the toLocalFile thing should have been) to provide that object. That way tooling like the CDT can use this new interface and still get the benefits from using File and avoid having to do weird things with the URI. The EFS interface as we have it now is intended to support pluging in file systems to support IResource. Changing it's interface changes that and I don't think we want to do that. It's a pretty basic API design principle to keep a separation of concerns and the new interface would do that. I think the adapter version of Chris's proposal accomplishes much of these objectives. I just want to make sure we can use this interface as a replacement for File and URIs when creating paths for interacting with external tools.
(In reply to comment #21) > I'm curious to hear what the other e4 resources folks think... to me it feels > strange to be pushing the logical IResource structure down to the EFS level. > This could be supported by having separate links at the IResource level for each > place where the resource tree structure diverges from the physical file system > structure. This would allow EFS to map only to the physical structure. > The problem is though, by not putting strict restrictions on this previously, and by populating the contents as reported by EFS directly into the IResource tree, by default the contract is now that EFS represents what's in the resource tree. This is why for example Doug has in the past tried faking out the resource system using EFS to create a virtual file system. > I can understand the use case of a zip/tar being expanded in place to become > part of the file tree, but that by itself doesn't require introducing > logical/physical parent/child concepts. I think all we would need for that > particular case is the API described in comment #0 - a method to get the > physical path of an IFileStore. Well, after more thought, I think I agree with one thing, which is that it's not possible for a store to have both a virtual child and a physical child at the same time. That would mean, for example, that you have virtual folders, that also exist on disk? Doesn't make sense. So maybe we only need one method to get children. Parent relationships can be both virtual or physical though, because individual files can have both a virtual folder parent in the resource system, and a physical parent on disk. I don't really want to rely on path manipulation for parent/child relationships. Firstly, it's inconvenient. By the same logic you mention, in theory then all anyone needs on IFilestore is getPath() and we don't need parent/child methods at all. I don't think anyone thinks that's the case. Secondly though, there is a huge problem right now with trying to manipulate paths which are not in the format of the local machine. See bug 207078 for some of the discussion on that. IPath is currently ill-equipped right now to say construct and manipulate paths for a remote Windows machine when my local machine running Eclipse is UNIX-based.
(In reply to comment #22) > What I think we need is really a remote capable version of java.io.File. We > could write a Local implementation that delegates to File, and we can provide > various remote implementations that use things like ChannelSftp from jsch. We > could then provide a toPhysicalFile() or an adapter (which is probably what the > toLocalFile thing should have been) to provide that object. That way tooling > like the CDT can use this new interface and still get the benefits from using > File and avoid having to do weird things with the URI. > So are you proposing we would literally subclass java.io.File? From a raw functionality point of view it's fine, but the thing I have against that is that then we lose things like progress reporting and cancelability, which was part of the reason I assume that IFileStore and friends were created in the first place rather than just having everyone subclass File. > The EFS interface as we have it now is intended to support pluging in file > systems to support IResource. Changing it's interface changes that and I don't > think we want to do that. It's a pretty basic API design principle to keep a > separation of concerns and the new interface would do that. > > I think the adapter version of Chris's proposal accomplishes much of these > objectives. I just want to make sure we can use this interface as a replacement > for File and URIs when creating paths for interacting with external tools. As far as separation of concerns goes, yes, the adapter approach is better for that.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.
