RE: [cdt-dev] Performance issues with indexer/refresh on large scale pr

[<Tim.Kelly@xxxxxxxxx>]

Some things we found when doing performance analysis/benchmarking in a similar scenario (the general case for Nokia developers):

 

1) Import time is linear WRT # of source files under the root.

After about 15K source files under the root it took a couple minutes to import a project.

At about 25K source files is took almost 5 minutes. We decided import was the time you click Finish on a project creation wizard until you can open a source file from the Project Explorer.

We added a timer to our project creation action posted the project creation time to the error log the same way the Indexer does (i.e. Indexed '<project>' ....) so we could monitor problematic projects.

The majority of time importing is spent querying the file system for file attribs & directory sublists (refreshing is actually quite fast though to the user it appears to be a a lingering job)

 

2) Indexing time is related to # of file under the source roots. We introduced the concept of nested source roots but I see the target milestone was set to 7.0.

See bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=181020. The most extreme case we got indexing down from 324 seconds to 27 seconds by addressing nested source paths.

 

However, for you case of 40K source files, I think you will need e4 (as Doug already mentioned). I found any project with that many source files under the root makes the IDE about unusable.

 

Tim

---

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of ext Doug Schaefer
Sent: Tuesday, January 26, 2010 9:23 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] Performance issues with indexer/refresh on large scale project

On Tue, Jan 26, 2010 at 10:07 AM, Bursian, Achim <achim.bursian@xxxxxxxxxxx> wrote:

Hi all,

I'm running into serious performance problems with a ~40k files C++ project (standard make project on Solaris/Sparc with ClearCase as SCC). Importing and indexing the project takes hours, which is at least partly caused by a lot of unnecessary files being opened.

 

Three questions:

* How can I programmatically disable the automatic indexing for a period of time, and reenable it later (I found that it's faster to do a refresh without the indexer running, and doing a full indexer run afterwards)? I'd try to do that in a startup plugin, which would disable indexing, import the project, refresh everything, then run the indexer. Maybe that could speed up things...

 

* I would like to exclude a complete part of the project hierarchy from being seen by Eclipse and CDT. I thought the "filter" setting ("Source folder exclusion pattern") on the CDT source folders property page would do just that, but no. I excluded the "test/" folder and some others with the filter, now it is no longer shown in the project explorer. But monitoring the system with Solaris dtrace shows that the whole <projectroot>/test folder hierarchy is still scanned (stat system call) during a refresh, and a lot of files there even get opened (Imakefiles, *.txt, *.xml). This is quite bad, as it contains 20k files that are only testcases and data stuff. Anything I can do?

This is actually the Eclipse Platform refreshing your project, not the CDT. This is one of the main drivers for the Flexible Resources feature coming in this year's train release, to cut down on what Eclipse sees in a project directory tree.
 

* Is there any general guideline for setting up such a large scale project with CDT, anywhere?

 

Thanks!

  Achim

 

 

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev