Bug 310514 - Support exporting a Resource Snapshot along with a CDT shared index
Summary: Support exporting a Resource Snapshot along with a CDT shared index
Status: RESOLVED FIXED
Alias: None
Product: CDT
Classification: Tools
Component: cdt-indexer (show other bugs)
Version: 7.0   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: 7.0   Edit
Assignee: Markus Schorn CLA
QA Contact: Markus Schorn CLA
URL:
Whiteboard:
Keywords: contributed
Depends on: 306575
Blocks: 259280
  Show dependency tree
 
Reported: 2010-04-26 14:03 EDT by Martin Oberhuber CLA
Modified: 2010-07-28 15:23 EDT (History)
3 users (show)

See Also:

Attachments
patch v1 (11.23 KB, patch)
2010-04-26 14:06 EDT, Martin Oberhuber CLA 		mschorn.eclipse: iplog+
Details | Diff
updated patch (16.87 KB, patch)
2010-04-27 08:35 EDT, Markus Schorn CLA 		mschorn.eclipse: iplog-
Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Oberhuber CLA 2010-04-26 14:03:52 EDT 
+++ This bug was initially created as a clone of Bug #306575 +++

One problem with using the CDT team-shared index on a REALLY large source base (like 100000 files) is that when importing a project on a slow file system such as a ClearCase dynamic view, most of the time during import is spent for doing an fstat on each file to find out its timestamp.

As a provider of a large source code base, I want to provide this source code along with a pre-computed refresh snapshot, such that people importing my source base can get access to the project much faster.

Eclipse Platform M7 is going to provide support for this, and the CDT team-shared index export wizard should leverage it.
Comment 1 Martin Oberhuber CLA 2010-04-26 14:06:38 EDT 
Created attachment 166112 [details]
patch v1

Attached is a first simple patch providing the feature. 

I consider this functionally complete, though it could have some polishing (e.g. add a "Variables" button for defining custom Path Variables in the wizard when needed, or changing the way how the TeamPDOMImportOperation#expandLocation() code is called.

A first test of the feature on a medium-sized project shows that the feature can cut down project import time (that is, time until productivity) from 1 minute to 5 seconds.

Comments welcome.
Comment 2 Martin Oberhuber CLA 2010-04-26 14:08:38 EDT 
I think that this feature also implements bug 259280, at least partially.
Comment 3 Markus Schorn CLA 2010-04-27 08:35:57 EDT 
Created attachment 166189 [details]
updated patch

Thanks Martin,
I have made the following modifications to the patch:
* Removed feedback about the export location for the resources snapshot in order
  to avoid variable expansions during the verification. (This could bring up
  modal dialogs)
* Adjusted the name-computation for the resources snapshot file.
* Externalized strings.
Comment 4 Markus Schorn CLA 2010-04-27 08:37:33 EDT 
Implemented in 7.0 > 20100427.
Comment 5 Francis Lynch CLA 2010-04-28 08:38:42 EDT 
(In reply to comment #2)
> I think that this feature also implements bug 259280, at least partially.
This patch certainly implements a big part of the "project migration" feature described in 259280, but there is a very important part that still remains. That is the ability to specify a "trusted" team shared index, which is one that does not require checksum validation of the files referenced in the imported PDOM. Without this, the user scenario of creating a new project for a new ClearCase view based on a ClearCase "reference" view does not gain the desired performance improvement. For the customer I have been working with, the PDOM file size is just over 2 Gb, and recomputing the checksums and index for all 50000+ files it references requires several hours. Our custom implementation for this customer produces a team shared index zip file that does not include the checksums so that the import "trusts" the shared index to be an accurate representation of the files in the imported project. I'd like to invite comments on whether it would be possible to provide this option (or something similar) with this feature as well.
Comment 6 Markus Schorn CLA 2010-04-28 10:18:47 EDT 
(In reply to comment #5)
I have moved the discussion to bug 259280.