Community
Participate
Working Groups
The Platform/Team Target Management component including its FTP and WebDAV connectors is being discontinued as of Eclipse 3.3: http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01040.html Target Management / RSE is the logical successor of this functionality. We should therefore create a bridge from RSE IFileService to the Platform/Team synchronization APIs, such that the synchronization functionality previously offered by Platform/Team becomes available through RSE. This synchronization framework is appreciated by the community: http://dev.eclipse.org/mhonarc/lists/platform-team-dev/msg00229.html Creating the bridge will likely happen after TM 2.0; community help would be appreciated. See also bug #185921.
*** Bug 202904 has been marked as a duplicate of this bug. ***
*** Bug 203001 also asked for synchronization of remote folder structures. In terms of UI, I could imagine that in the RSE System View, user selects any folder and chooses right-click > Synchronize with... A dialog would allow picking the other folder to synchronize with. Given that information (source and destination), the Synchronization Perspective should open to do the actual sync. RSE IFileService would simply be the provider giving the input. The current Platform/Team Extras also provides File>Import and File>Export wizards which make use of the Synchronization APIs. RSE currently provides these through importexport wizard without using synchronization. One advantage of the RSE solution is that the import/export settings can be stored to a file for later re-use; one disadvantage is that it does dumb copy instead of synchronization. I believe that ideally, RSE importexport should also use the Team/Synchronization Framework.
What's the status here? Synchronization is a feature I want badly enough to help with it's migration/development. Somewhere along the way the FTP/WebDAV synchronization provided Platform/Team Target Management went from decrepit to broken, and I am kind of at a loss for the functionality. Thanks for the updates.
(In reply to comment #3) > What's the status here? Synchronization is a feature I want badly enough to > help with it's migration/development. Somewhere along the way the FTP/WebDAV > synchronization provided Platform/Team Target Management went from decrepit to > broken, and I am kind of at a loss for the functionality. > > Thanks for the updates. > Same here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=214467 Did you find any workaround for this or using external tools for synchronize? I hope this functionality comes back soon - thank you
Thanks for offering help, Jason. I guess current status of this bug is that everybody wants it, but nobody has looked into how to actually get it done so far. Any help is appreciated. I'd propose to start with a description of requirements and proposed user experience, then look at APIs and how to get it done.
Valter: As far as external tools, I've coped in several different ways. I have converted some projects to a CVS store and used Eclipse's excellent CVS functionality to synchronize them. Most web applications, however, are not worth the trouble of trying to CVS them, and this is the context in which I most frequently used FTP synchronization. Now, I mostly use the RSE to open files I wish to edit, and let the RSE save them remotely so there is no more local copy. Slower and less convenient, but it does keep the resource in sync. Martin: I delved a bit into the Team/Target API when I was looking for the NullPointerException that eventually killed my usage of the old Team synchronization, so I at least have a generalized view of it. I can't promise too much because I don't want to overextend myself, but I will work on some requirements and proposed user experience. Is there a location these documents should be posted? Here? Thanks!
Yes, please attach any related documents on this bug. You might also find the discussions about remote folder compare interesting on bug 203001, since they are related to the Platform/Team APIs.
Created attachment 86907 [details] C-Requirements for TM Sync View Here's a first draft of some C-requirements. I'll work on some D-requirements too if you would like, and I'll get on the user experience too. Either let me know or post any changes that need to be made.
Thanks for your requirements doc. It looks to me like if the current RSE Import/Export wizard already provides everything as requested for initial synchronization, so it's only missing on incremental synchronization afterwards. In order to facilitate that, it looks like the following is missing: * Remember timestamp of the last synchronization * Integrate with Team/Synchronize view, based on RSE import/export settings Here are some other comments: 1. It seems to me that the act of importing/exporting/synchronizing with a remote file system is not specific to RSE, but could also be implemented through any registered EFS provider. That would make the whole thing more versatile. I wonder if the Platform team ever thought about that? Even synchronization between two directories on the same local system could be provided through that. I'd assume that it should be relatively easy to re-write the current RSE import/export wizards such that not RSE but EFS is used as the underlying transport. DaveM can you confirm? 2. I do not like the idea of allowing an RSE Filter as the remote endpoint. For synchronization, directory structures should match on the local and remote side. Therefore, only folders should be allowed; if filtering is desired, that should be set up in the import/export wizard (and ONLY there). 3. When a project / folder is already shared with a CM system, I assume that we'd still want our "folder synchronization" to work. Therefore, we cannot use Team > Synchronize for that. What would you think about providing the "Synchronize" operation in the context menu of the *.rexp file that RSE can create storing the settings of a particular synchronization? - That would even allow multiple different synchronizations (to different hosts for instance) of the same project or folder. Or would the feature be too hidden if implemented like that? What other ways could we use to give the user access to the actual synchronize operation? 4. Regarding some other small points from your requirements: * Time difference between local and remote system -- couldn't that be computed automatically? E.g. when the first file is transferred, compare the local timestamp with the remote one? * Is it really sufficient to always do a dumb copy-all-overwrite on the initial import/export? Or do we need to be smarter, e.g. for a case where a previous workspace with existing sync info has been lost? * For "ignored resources" I think that we should ONLY take the settings from the import/export wizard. The wizard could have a checkbox for "Also use filter from Preferences/Team" to make that explicit.
(1) Are you planning to decorate the local files that are modified (relative to tehe time you made synchronization)? If the answer is yes, than you will have all kind of problems with projects that are already shared with CVS/SVN (they also contain a decoration for modified files, so using 2 decorations becomes confusing for the end-user) (2) what is the behavior when i check-out my project from CVS and then I create a "connection" between the shared project (in eclipse) and a remote location? Do you perform any operation on the remote location? The remote location might contain the files, and you can't know which version of the files (if you are based only on the timestamps) so you start with an "unstable" state.
(In reply to comment #10) Here's just my thoughts, until Takuya gets up to speed on this: (1) Decorations can be selected by users (Window > Preferences > General > Appearance > Label Decorations). I see no problem with multiple decorations if the user wants to have them. They might be disabled by default though. (2) Initial synchronization cannot be incremental. What you'd see is just like what you see in a Compare/Merge tool. Depending on how far Takuya want to go, I could even imagine creating an MD5 hash for each file locally and remote, and treat those the same that have same size and hash. The hash algorithm could even be pluggable. I don't think that the initial implementation will have label decorators -- it will work more like the "old" FTP-WebDAV plugin from http://archive.eclipse.org/eclipse/downloads/drops/R-3.2.2-200702121330/download.php?dropFile=eclipse-FTP-WebDAV-3.2.2.zip with its description on http://dev.eclipse.org/viewcvs/index.cgi/platform-vcm-home/plugins/target/target-project-sets/readme.html?view=co and its CVS Team Project Set (*.psf) on http://dev.eclipse.org/viewcvs/index.cgi/platform-vcm-home/plugins/target/target-project-sets/all-target.psf?view=co
Some good Platform docs for this are on http://help.eclipse.org/help33/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/team_synchronize_beyond_basics.htm
Great to hear that there will be a Summer of Code student on this. I'm still using the Team/Extras features in order to have things show up in the Synchronize view, and would be happy to be an early adopter of this effort.
For everybody's information, with the old Platform Team/Extras plugin, the Synchronize Perspective can be used for FTP or WebDAV synchronizations as follows: 1. Switch to Synchronize perspective 2. In the Synchronize view, use the top-left drop-down and hit “Synchronize…” 3. If you have the Team/Extras stuff installed you will see FTP and WebDav under “Available synchronizations”, so just select that. 4. Click through the wizard and you should be all set. Trying that out, leaves one important question to answer: * How to treat Linked Resources in the RSE Synchronization Provider? It turns out that FTP from Team/Extras does handle linked resource folders in the workspace as real directories on the remote. Other providers like CVS always ignore linked resources. What should the RSE provider do? (A) Handling Linked Resources like Folders does provide some added versatility, since it allows for masking or redirecting parts of the remote tree in the workspace. Also, plain "Workspace" synchronizations can be extended to "outside the workspace" resources by means of linked resources. (B) On the other hand, liked resources can change the file system structure and requires special handling of workspace resources compared to non-workspace resources. This means more potential confusion, and requirement for a special implementation just to handle workspace resources, where otherwise everything could be treated as EFS IFileStore. Does anybody have an opinion in favor of (A) handling linked resources as folders versus (B) ignoring linked resources during synchronize?
On second thought, there is a third option: (C) In the synchronize wizard, allow the user to specify how linked resources should be handled (ignore, or map to folder on export; ignore, or map to redirection on import). Given that we want to separate the UI part of synchronize from a non-UI API, it looks like a pragmatic approach is to specify the API such that it can handle "Workspace Resources" as well as "File System Resources", and such that it allows for an option to ignore or follow Linked Resources. Once that API is specified, an initial implementation could be done to always ignore the linked resources, but such that the API is open for later extension that is capable of observing (following) linked resources. One drawback of allowing Workspace Resources in the synchronize API is, that an additional independent API is required for allowing remote-to-remote folder comparisons that do not involve the workspace (as per bug 203001). In the end, the synchronize API should be simpler than the current RSE RemoteFileTransferUtility. An API could look like this: interface RSESynchronizer { public void synchronize(IContainer source, IFileStore target, IFileSystemFilter filter, Calendar lastSyncDate, int options); public void synchronize(IFileStore source, IFileStore target, IFileSystemFilter filter, Calendar lastSyncDate, int options); } Where options would include values such as SYNC_MODE_OVERRIDE_SOURCE = 1; /** "Import": Override source with destin. */ SYNC_MODE_OVERRIDE_DEST = 2; /** "Export": Override destin. with source */ SYNC_MODE_OVERRIDE_OLDER = 3; /** Override older files with newer ones */ SYNC_MODE_DELETE_MISSING = 4; /** Delete missing files */ SYNC_MODE_FOLLOW_LINKS = 8; /** Follow linked resources */ SYNC_MODE_UI_REVIEW = 16; /** Review sync in UI */ which are similar to the command-line options that tools like rsync provide. In order to avoid the somewhat ugly distinction between IContainer vs. IFileStore (vs. probably other formats of virtual file systems), we could also abstract the concept of file system container and provide a hierarchy of implementations like this: interface IFileSystemContainer class WorkspaceFileSystemWithoutLinkedResources class WorkspaceFileSystemWithLinkedResources class EFSFileSystemContainer class RSEVirtualFileSystemContainer this would allow yet more flexible extension of the synchronize implementation in the future, allowing e.g. to synchronize ZIP file contents with file systems.
Created attachment 107408 [details] mid-term deliverable This is the mid-term deliverable of GSoC. If you are interested in this, feel free to try it. About how to use, please refer to the other demo script file.
Created attachment 107488 [details] demo script for mid-term deliverable
Created attachment 107797 [details] excutable mid-term deliverable I also attached demo script for mid-term deliverable I've already attached. If you would like to test, you need some preparations. * Eclipse 3.4.0 * Target Management (RSE) 3.0 * Remote host * Dstore (rseserver) or SSH * Eclipse 3.4.0 I develope and test in Eclipse 3.4.0, and please use it. You can find Eclipse 3.4.0 in the page (http://www.eclipse.org/downloads/). * Target Management (RSE) 3.0 plugin My project use TM (RSE) 3.0, so you need to get the plugin. You can install the TM 3.0 plugin from update site (http://download.eclipse.org/dsdp/tm/updates/3.0/) or zip (http://download.eclipse.org/dsdp/tm/downloads/drops/R-3.0-200806202130/) More detail about RSE, please visit http://www.eclipse.org/dsdp/tm/ * Remote host RSE manage resources between local and remote host transparently. So, you need to have remote host. VirtualBox is ok. As the demo script, I tested in VirtualBox. * Dstore(rseserrver) or SSH I use the dstore or SSH as the protocol in RSE. If you have already ssh server, you don't need to anything about this. If you would like to use dsotre, you can get from here(http://download.eclipse.org/dsdp/tm/downloads/drops/R-3.0-200806202130/index.php) Please download the rseserver-3.0-(your platform) Installing the mid-term deliverable. For installation, you simply download my mid-term deliverable named as "org.eclipse.rse.synchronize_0.1.0.jar". and put it in the plugin directory of your eclipse. Then, you need to restart Eclipse and enjoy it. In the demo script, I use dstore as protocol. If you use ssh, please read the document substituting ssh for dstore
Nice work Takuya Miyamoto! I'm unable to find your instructions for using this with SSH, and so far the RSE FTP does not appear to function with this. Could you direct me to the document on making this function via SSH. Also, does this duplicate work already done in the Team Synchronization perspective? It would seem like a natural solution to include this as a provider listed in the Team Sync view. Thanks for your hard work
(In reply to comment #19) Thank you for your comment, Mark! > I'm unable to find your instructions for using this with SSH, and so far the > RSE FTP does not appear to function with this. Could you direct me to the > document on making this function via SSH. If you would like to use with ssh, you need to make ssh connection in RSE. After that, you can use on the ssh connection in the Remote System Explorer as the same to dstore in my demo script. And, please use the "excutable mid-term deliverable". I took a mistake to attache "mid-term deliverable" that contains only source code so it doesn't work. > Also, does this duplicate work already done in the Team Synchronization > perspective? It would seem like a natural solution to include this as a > provider listed in the Team Sync view. I see. I didn't think such function. I consider it. Thanks.
(In reply to comment #20) > > Also, does this duplicate work already done in the Team Synchronization Just to clarify, the RSE Synchronize Wizard does open the Team Synchronization perspective in order to perform the actual UI related tasks of synchronization, so it is integrated. What's missing currently, is to make RSE a synchronize provider that's exposed in the UI: Under Window > Open Perspective > Team Synchronization, select the "Synchronize" view and click the little dropdown arrow beside the synchronize icon in the view-local toolbar. With the RSE synchronize plugin installed, it should list a "Synchronize Remote Resources (RSE)" provider but it's not. The relevant extension is currently not active in plugin.xml, it looks like the "org.eclipse.team.ui.synchronizeWizards" extension point must be implemented.
Martin: when this is ready for beta testing let me know, as I'd be happy to update my workspace to use it.
Created attachment 110113 [details] Demo script Now, update site is available. http://eclipse-incub.sourceforge.net/updates-soc/rse-sync/ Additionally, I attached demo script for it. It explains how to install and use in detail. If you would like to test it, please read my demo script at the beginning. Feel free to add your comment on this Bugzilla or Wiki (http://wiki.eclipse.org/Platform/Team_Synchronization_on_top_of_RSE).
Created attachment 113458 [details] Snapshot of plugins at end of SOC Summer of code is over, and an alpha version of the plugin is available. See http://wiki.eclipse.org/Platform/Team_Synchronization_on_top_of_RSE#Current_Status for the current status, what works or not, and how to continue. We do intend to continue work on the plugin, and we appreciate community feedback, ideas and patches. The code is currently hosted on Sourceforge, and I'll attach team project sets for getting going. See the Wiki mentioned above for more details.
Created attachment 113460 [details] Project set for anonymous (read-only) access to sourceforge For anonymous (read-only) access to the projects on sourceforge, save attached *.psf file to a local disk, then File > Import > Team > Project Set.
Created attachment 113461 [details] Project set for sourceforge committer (write) access For sourceforge committer (write) access to the projects on CVS, save attached *.psf file to a local disk, then File > Import > Team > Project Set.
*** Bug 248556 has been marked as a duplicate of this bug. ***
I filed IPZilla https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2714 asking for a legal review to get the code moved into the main Eclipse.org Repository. DaveM: The idea of the plugin is to add an additional "review" checkbox to the existing import/export wizard. When this checkbox is ticked, the Team/Synchronization perspective will open for review. If not, the old mechanism is used. If we merge this into the current rse importexport plugin, we'll add a dependency from importexport to org.eclipse.team.core, org.eclipse.team.ui, org.eclipse.compare which is not currently there. Do you think that this is in order, or should we investigate means to keep the Team dependency out of importexport? The compare dependency should not be an issue, since rse.files.ui already depends on compare.
Again, Community contributions to this are very welcome. I don't think there's too much missing to make this really fly. See the attachment 110113 [details] demo script for how it works. The main issue to fix will be getting the diff state right and the menu items in the synchronize view right. Comparing the plugin source against the old FTP/WebDAV implementation should help getting this fixed (see comment 11 for how to get that).
(In reply to comment #28) > I filed IPZilla https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2714 asking for > a legal review to get the code moved into the main Eclipse.org Repository. > > DaveM: The idea of the plugin is to add an additional "review" checkbox to the > existing import/export wizard. When this checkbox is ticked, the > Team/Synchronization perspective will open for review. If not, the old > mechanism is used. > > If we merge this into the current rse importexport plugin, we'll add a > dependency from importexport to > org.eclipse.team.core, > org.eclipse.team.ui, > org.eclipse.compare > which is not currently there. Do you think that this is in order, or should we > investigate means to keep the Team dependency out of importexport? The compare > dependency should not be an issue, since rse.files.ui already depends on > compare. > As you say, RSE already brings in org.eclipse.compare with it's files.ui plugin and I don't think the team stuff is too heavy weight, so I'm in favour of bringing this stuff in to enhance import/export.
Legal Message: I, Takuya Miyamoto, declare that I developed attached code from scratch, without referencing any 3rd party materials except material licensed under the EPL. {I am authorized by my employer to make this contribution under the EPL.}
(In reply to comment #0) > The Platform/Team Target Management component including its FTP and WebDAV > connectors is being discontinued as of Eclipse 3.3: > http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01040.html > > Target Management / RSE is the logical successor of this functionality. > > We should therefore create a bridge from RSE IFileService to the Platform/Team > synchronization APIs, such that the synchronization functionality previously > offered by Platform/Team becomes available through RSE. This synchronization > framework is appreciated by the community: > http://dev.eclipse.org/mhonarc/lists/platform-team-dev/msg00229.html > > Creating the bridge will likely happen after TM 2.0; community help would be > appreciated. See also bug #185921. > Did you see an SFTP plugin for eclipse... http://www.jcraft.com/eclipse-sftp/
I confirm that I am not employed and own the Copyright of the code that I wrote.
Got IPZilla approval, will merge into RSE 3.1M4.
The initial summer-of-code contribution is now merged into RSE HEAD (3.1M5) It is now in the org.eclipse.rse.importexport plugin, with testcases in the org.eclipse.rse.tests plugin. The status of the contribution is still "alpha", not yet ready for general use. For details, see the Wiki page on http://wiki.eclipse.org/RSESync/Status_and_Ideas I'm still closing this bug since the original contribution is in. We'll use separate bugs to track remaining issues. (In reply to comment #32) > Did you see an SFTP plugin for eclipse... > http://www.jcraft.com/eclipse-sftp/ The SFTP team provider leverages the "org.eclipse.target" infrastructure that used to be part of the Team/Extras download. This infrastructure has been deprecated and removed from Eclipse downloads in the Europa (Eclipse 3.3) time frame. It does continue to work, but it's unsure for how long. Therefore, we're going with the RSE infrastructure for this now, and may be looking at EFS / e4 Resources in the future.
(In reply to comment #35) > The initial summer-of-code contribution is now merged into RSE HEAD (3.1M5) Martin, it would be great for you to blog about this GSOC success when the time is right. I've been trying to get more projects to post ideas... http://wiki.eclipse.org/Google_Summer_of_Code_2009_Ideas
Comment on attachment 107797 [details] excutable mid-term deliverable Marking iplog+ to give credit to the contributor. The full IP Review is on https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2714
