Community
Participate
Working Groups
+++ This bug was initially created as a clone of Bug #301563 +++ User Story: ----------- When importing a large project into Eclipse today, a deep refresh is started to populate the resource tree. This deep refresh takes long on a slow file system (in the range of minutes), and while this refresh is ongoing, quite many UI actions can not be performed. As a developer, I need to start working on a large imported project quickly. Therefore, I'd like to avoid this deep refresh and populate my resource tree from a pre-built snapshot instead. Any discrepancies between the snapshot and the actual file system can then be resolved as a low-priority refresh process in background. Constraints: ------------ With bug 301563, we have created a core mechanism for exporting and importing resource tree info. This is already in use today through domain-specific wizards for export and import: Essentially, clients of the Platform can write a UI-less application to create the resource tree snapshot, as well as a domain-specific wizard to import a project with snapshot. We'd now like to provide access to this functionality in the base Eclipse Platform, under the following constraints: - New UI must be non-intrusive since we expect only few users (with very large projects) to actually need the functionality, compared to the full installed base of the Eclipse IDE. - Various Producer-Consumer scenarios need to be considered (who creates the snapshot, where does it live, when is it updated, and how is it consumed). In many cases, this functionality will be related to a "Team" setup of some sort, with a team prime creating the snapshot and users importing it. - Potential extenders of the snapshot functionality are any kinds of indexers. The CDT team-shared index is very related to this. A JDT index could also be related. That much for the problem description. I'll post some implementation ideas next.
I think that the main problem to solve is the "Import" problem: How would the "project-open" operation know where to look for the snapshot data on import. Once we agree on how we want to "import" the snapshots, it will be easier to define a way how snaphots are to be created. At the moment, I can think of the following options for making the "import from resource snapshot" functionality available to base Eclipse: 1. Project-level Preference. ---------------------------- A Project Preference points to a location where the IProject#open() method is expected to look for a snapshot. If it's found, it's automatically used, so there is no UI needed for "import". Creating the snapshot is done through a wizard similar to the CDT team-shared one, or from a Project Preference page where I can both enter the location and press an "Update" button to update the snapshot. 2. Enhancement to the existing "Import Existing Project" wizard --------------------------------------------------------------- Add a field "Snapshot Location" to the existing "Import Existing Projects" wizard. For each project to import, the wizard would search the snapshot location for any meta-data that should be unpersisted at the same time when importing the project. A new "Export Snapshots" wizard would populate such a metadata location. 3. Provide a new "Import existing Workspace" wizard --------------------------------------------------- Since we are importing project reference + meta-info about the project, it would make sense to have a new "Import existing Workspace" wizard to do that. An "Export Workspace" wizard could be its counterpart. The "exported workspace" would consist of project references + the snapshot metadata. 4. Global Preference. --------------------- Define a global Preference providing a location where the import wizard will look for metadata to populate on an import. Other ideas are welcome!!
I personally like the "Project-level Preference" option as a start. While it may not solve all scenarios, this does solve the scanario where the project content (producer) essentially "owns" the snapshot. UI could be very non-intrusive, and the pattern has successfully been rolled out with the CDT team-shared index. Additional functionality (where the project does not "own" its snapshot) could be added later on.
Created attachment 161137 [details] Patch to org.eclipse.ui.ide with simple import/export wizards Here is a patch with simple refresh snapshot export and import wizards with limited capabilities but useful for testing with real projects. The snapshot is written to exportFolder/<projectName>-index.zip and only local folders are currently supported. On import, only the current folder is scanned for projects in order to avoid potential performance problems on slow file systems.
(In reply to comment #3) I'm not sure how the Import wizard should be used. When I fill the 'Select source directory' field, the viewer below is not populated. What is expected to be in 'Source directory'?
Created attachment 161436 [details] The viewer is not populated
(In reply to comment #5) > Created an attachment (id=161436) [details] > The viewer is not populated I think what you are seeing here is happening because the import only looks in the current folder for projects. This is not symmetrical with the way export works, but we felt it was important not to automatically recurse through the subfolders because the import will typically be used on slow file systems where that recursion (especially for heavily nested source directories) could take a very long time. So if you navigate to a folder in your workspace where projects are defined, you should see the viewer get populated.
Thanks Francis. From Szymon's question I see that this is a real usability issue. I had not considered this yet since the feature was meant for "experts", but now I see that this was not a good idea after all. It may be better to just live with the default "Import Project" search taking long. As long as it is cancelable this might not be so much of an issue. We should shoot for keeping things as simple as possible for now. Addressing the "search for projects takes long" issue should probably be covered with a separate item.
(In reply to comment #7) > It may be better to just live with the default "Import Project" search taking > long. As long as it is cancelable this might not be so much of an issue. Might I suggest that we add a check box such as "Do not recurse into subfolders" that is unchecked by default? Then we could preserve the same behavior as in export but allow users on slow file systems to avoid the recursion overhead on import if desired.
I talked with Francis about it on the Resources call. It seems that confusing part is the description in the Snapshot Import wizard, i.e. "Select a directory to search for existing Eclipse projects". For me it means that I have to select a location containing project folders, while it is about a location with .project files. Another approach would be to select a source directory and if .project can't be found, the wizard would ask to scan recursively. Or maybe the project viewer should say 'click to scan recursively'. I think that we should make it clear to users, what is expected to be in the source directory anyway. Right now, we can Browse only the local file system for .project files. I think that it would be useful to also browse other kinds of FS. Some issues with the Export wizard. 1) Select more than one project. Open 'Snapshot Export Wizard' and 'Browse' for the snapshot directory. When a directory is chosen, Finish becomes disabled. 2) No prompt that we can override existing files Some other issues with the Import wizard. 1) No description when 'Project Refresh Snapshot' is selected in the Import wizard (the one where you can select a wizard to use) 2) I think it would be useful to show somehow, for which projects we have snapshots. 3) I think that error handling could be better. For instance, if there is no matching snapshot for the selected project, we get an error dialog, however the project is created in the workspace. I would expect the wizard to prevent the issue.
Any news? I guess Francis is on it. Martin, is that right? We had a chat with Martin that the UI is domain specific, so it may be hard to come up with something general. However I would like to see my concerns addressed in the next patch. Then we may discuss whether make it part of Eclipse SDK or example plug-ins. In any case, the UI should be rady by M7, so please mark the bug appropriately.
I talked to Francis, and he thinks that addressing your concerns in a new patch is worth about a day of work. However, addressing bug 305718 and bug 306575 is way more important for us than this UI, and I believe that these two are also much more promising in terms of providing something that's generically adoptable. Francis already attached a patch on bug 305718, and your initial review on the approach would be highly appreciated such that we can commit a solution. Thanks!
(In reply to comment #11) > I talked to Francis, and he thinks that addressing your concerns in a new patch > is worth about a day of work. Sounds good. > However, addressing bug 305718 and bug 306575 is way more important for us than > this UI, and I believe that these two are also much more promising in terms of > providing something that's generically adoptable. For me they are equally important. The fact that we have been thinking about having UI only in the examples doesn't mean that we give it up. > Francis already attached a patch on bug 305718, and your initial review on the > approach would be highly appreciated such that we can commit a solution. Thanks! I'll look at it.
(In reply to comment #12) > > Francis already attached a patch on bug 305718, and your initial review on the > > approach would be highly appreciated such that we can commit a solution. > Thanks! > > I'll look at it. I already commented on bug bug 305718. The patch looks quite good. UI would help to play with the fix a bit ;-) Any news on that?
Created attachment 164917 [details] Patch version 2 org.eclipse.ui.ide with import/export wizards Here is a patch that addresses some of the concerns raised by Szymon in comment #9. Export wizard fixes: (1) can now select more than one project; (2) generates prompt about overwriting existing files. Import wizard fixes: (1) added description for refresh snapshot in Import wizard dialog; (2) changed description to indicate that .project files are being searched for. After projects being imported are selected, the wizard now checks for the presence of a refresh snapshot in the directory and prompts the user if not present: No refresh snapshot found. Import project anyway? As an alternative we could modify the wizard not to display the project in the selection box if the snapshot is not found; it already has the feature of not displaying a project if a project of the same name exists in the workspace it's being imported into. At present, the import wizard still only displays .project files in the current directory; it does not automatically scan subdirectories for projects. We can add this once we have defined how it should work (whether automatic or by clicking a button, etc.).
(In reply to comment #14) > As an alternative we could > modify the wizard not to display the project in the selection box if the > snapshot is not found; it already has the feature of not displaying a project if > a project of the same name exists in the workspace it's being imported into. Or we could just fade projects for which we can't find the snapshot. And disable checkboxes for these projects, so one will not be able to import them. Or we can allow to select them, but when such a project is checked, we can show a warning (in the header) that some projects to import have no corresponding snapshots. > At present, the import wizard still only displays .project files in the current > directory; it does not automatically scan subdirectories for projects. We can > add this once we have defined how it should work (whether automatic or by > clicking a button, etc.). It would help. I tried to import couple projects from an old workspace. I had to do it one by one. I think that we could add a button for now. In the header we should write that only the selected folder is scanned and that the button should be pressed to scan folders recursively. I think that this may be already too late to introduce this UI in SDK. I'll check how to make it part of the examples.
The underlying problem here is, how to associate projects with snapshots. Today, the wizard tries to enumerate projects, and then find matching snapshots. In order to avoid scanning the source tree (which is kludgy as we have seen), we should turn the logic around: enumerate snapshots in a given location, and then find matching projects. In order to facilitate this, we would need to have the export wizard not only export the snapshots, but also the original location of the projects that the snapshots were created from. Then the import wizard can look at the original locations. This allows for several interesting use cases directly: 1.) User A exports snapshots for read-only projects on a common shared file system --> User B can import from the same original locations 2.) User exports snapshots from workspace A and imports into workspace B --> Locations remain unchanged 3.) Clearcase dynamic views --> locations remain all in /vobs/xxx It gets a bit harder when user B sees the project on a different physical location (e.g. create snapshots from M:/myview/.. and import to M:/yourview/..) but even that case is manageable since the mapping from original paths into the new paths is simple. In order to associate the original export location with a snapshot, we might want to change the snapshot file format to include a header which holds the original snapshot location. That's not the only possible way doing this; but if we want to go this way we need to do it now. Another possible way is exporting an "myExportName.properties" file along with the snapshots. Essentially, such a properties file is very similar to a team project set (.psf). Perhaps we could even have snapshots associated with the actual .psf import in the long run? Other than that, I agree that the wizards can't go into the Helios SDK but should go into an examples project.
(In reply to comment #16) I thought we could just match the project name from .project and the name of the snapshot file. I think this would be enough, if this UI is not going to be part of SDK (for now).
My proposal from comment 16 was trying to address the issue that we don't want to drill down into the source tree to find projects since that costs time, but on the other hand we don't want to have every project browsed to manually. It also addresses the problem that for some projects, snapshots may be available but for others not. If the project locations are stored with the snapshots, we don't need to drill down into the source tree to detect projects, because we already know where the projects are. But of course implementing this proposal takes time. If you'd rather want to go with something quick just to try it out, well, then we could probably do a breath-first-search for projects down to some max level eg 2, or have a button to drill down one more level. But that won't help you if your projects are dispersed across multiple locations, e.g. have a workspace /view/myview/vobs/projectA /view/referenceview/vobs/projectB /local/workspace/projectC and you want to replicate that same 3 projects in a different workspace or for a different user, and import all projects with snapshot.
(In reply to comment #18) > My proposal from comment 16 was trying to address the issue that we don't want > to drill down ... I understand that. I think we should not be so strict about the performance now, since this is going to be only an example. > If the project locations are stored with the snapshots, we don't need to drill > down into the source tree to detect projects, because we already know where the > projects are. Right. It makes sense. If we have the location in the snapshot, we could just point out in the wizard the folder with snapshots and that's all. I like it.
Martin, Francis, could you please address the remaining issues? Moreover this is going to be a part of the SDK example plug-ins, so instead of a patch for the UI plug-ins, I would expect to get a separate example plug-in.
Where is the repository for examples? What name do you suggest? Can you suggest a sample example?
Details for UI examples repo: :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse module: www/eclipse/platform-ui/plugins You can already find some example plug-ins there. Plug-in names are something like org.eclipse.ui.examples.filesystem or org.eclipse.ui.ide.examples.markers. For our example plug-in I would suggest org.eclipse.ui.ide.examples.projectsnapshot. When the plug-in is added, we need to ask Kim M. to add the plug-in to the examples feature.
Created attachment 168661 [details] Patch version 3 org.clipse.ui.ide with import/export wizards Attached is a revised patch that fixes the import in two respects: (1) a checkbox is added to force populating the project list by recursing into subdirectories; and (2) the displayed project list does NOT include projects for which a correspondingly named refresh snapshot is not found in the specified snapshot directory. This is the simpler alternative of what was discussed in comment #15. If this solution is deemed unsatisfactory, I can provide a slightly more complicated solution that still displays the projects in the list but grays them out if no snapshot is found. (Note that projects that already exist in the workspace into which they are being imported are not listed either.)
(In reply to comment #23) Thanks Francis. > (2) the displayed project list does NOT include projects for > which a correspondingly named refresh snapshot is not found in the specified > snapshot directory. This is fine. I found just two minor issues. 1) When the source directory (for .project) is chosen and there are some .project inside, however the snapshot directory is not selected yet, I get "Some projects were hidden because they exist in the workspace directory". I would expect to see "Some projects to import have no corresponding snapshots" which is defined in Messages but is not consumed in the code. 2) So far the list is refreshed only when the checkbox state is changed. I think we could do a refresh on changes in the source and snapshot directory fields too.
(In reply to comment #24) > I would expect to see "Some projects to import have no corresponding snapshots" > which is defined in Messages but is not consumed in the code. I will make this change later and post a new patch. > 2) So far the list is refreshed only when the checkbox state is changed. I > think we could do a refresh on changes in the source and snapshot directory > fields too. In my tests, the project list got refreshed whenever I changed either the source or the snapshot directory. Looking at the patch, I see the code that should be doing this, so I'm not sure why it isn't working for you.
Created attachment 168941 [details] Patch version 4 org.eclipse.ui.ide with import/export wizards On further testing, I found an error in my refresh code: thanks, Szymon! Hopefully this version will work better when changing the refresh or projects directory. I also changed the message for hidden projects to be "Some projects were hidden because they exist in the workspace or have no refresh snapshots".
Thanks Francis. It looks good. The only change I would consider is to refresh the list and dialog buttons on each change in the snapshot and source fields. Right now, as I can see, we have to press Enter in order to refresh. Then we have to move the code to a separate example plug-in.
Created attachment 169077 [details] Zip containing the example plug-in Francis, I moved your work to a separate example plug-in. According to http://www.eclipse.org/legal/EclipseLegalProcessPoster.pdf, Martin O. can commit the plug-in to the repo, if he signed Member Committer Agreement. Otherwise we have to go through the full IP process. The right place for the plug-in is: :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse module: www/eclipse/platform-ui/plugins/org.eclipse.ui.ide.examples.projectsnapshot Martin, do you have time to look at it? I'm not sure, if you have commit rights in org.eclipse. If not, I'll start the IP process.
I have just found this https://dev.eclipse.org/ipzilla/show_bug.cgi?id=3773. Since the Project Snapshot Example plug-in derives from the code that was already approved, could we consider it approved too?
Created attachment 169110 [details] Patch version 5 org.eclipse.ui.ide with import/export wizards Szymon, here is a patch that corrects the problem you pointed out in the last comment, namely that the projects list should be refreshed on any change to the snapshot directory. I meant to make these changes and post this last night but didn't get to it until this morning. Perhaps you have already fixed this in your plugin, but if not, I think it should be fairly easy to merge the code from here into the plugin source. Thanks for all of your work on this.
Created attachment 169137 [details] Zip containing the example plug-in based on the patch version 5
Created attachment 169150 [details] A patch with Francis' changes for the newly created plug-in project in org.eclipse repo I have created the org.eclipse.ui.ide.examples.projectsnapshot module in the repository. It contains an empty plug-in project. This patch is against this project. CVS details for the example plug-in: :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse module: www/eclipse/platform-ui/plugins/org.eclipse.ui.ide.examples.projectsnapshot
CQ 4127 filed.
Francis, regarding the "Francis' changes against the newly created plug-in project in org.eclipse repo" patch - did you author 100% of it, or were there other contributors?
(In reply to comment #34) > Francis, regarding the "Francis' changes against the newly created plug-in > project in org.eclipse repo" patch - did you author 100% of it, or were there > other contributors? There were no other contributors.
The CQ is approved and I committed the patch. Please raise separate bug reports for remaining issues. Thanks Francis!
