Re: [cdt-dev] Memory usage during indexing

[Andrew Eidsness <eclipse@xxxxxxxxxx>]

Oops, I actually had one of my commits on top of master (the one in https://git.eclipse.org/r/#/c/19873).  Other than
that four-line patch I was based on e72c03:

commit e72c037dbd749499bb0ff5fb8a41bd68e42598b8
Author: Uwe Stieber <uwe.stieber@xxxxxxxxxxxxx>
Date:   Sun Dec 15 17:28:15 2013 +0100

    Bug 419391 - Contribute Windows PTY support

    Signed-off-by: Uwe Stieber <uwe.stieber@xxxxxxxxxxxxx>

-Andrew

On 13-12-17 02:31 PM, Marc-André Laperle wrote:
> Hi Andrew,
> 
> Which commit of master exactly did you use?
> 
> I tried this:
> git log 23c2fd --
> fatal: bad revision '23c2fd'
> 
> Was that after this commit?
> http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=3fbe0d12af6ebf1a570defb83b105ba8d53ce59b
> 
> Marc-Andre
> 
> 
> On 13-12-17 02:08 PM, Andrew Eidsness wrote:
>> Have other people been noticed that more memory is being used during indexing in recent builds?  I would expect some
>> extra memory and runtime usage since the parser/indexer are able to recognize more things.  However, the tests I've done
>> show only a small increase in what is found to put into the index at a cost of much more runtime and memory.
>>
>> We initially noticed the problem on internal projects, but I've been able to reproduce similar results using boost as
>> the test project.  In order to get repeatable results I've been using a project that I created from boost release tag
>> 1.55.0:
>>
>>     $ svn co http://svn.boost.org/svn/boost/tags/release/Boost_1_55_0 boost-1.55.0
>>
>> I imported this into Eclipse using "Import - Existing code as Makefile Project".  I choose only the boost folder (within
>> the boost-1.55.0 checkout folder).  Trying to index the complete source runs into significant problems when it gets to:
>>
>>     boost-1.55.0/libs/preprocessor/doc/example/delay.c
>>
>> Using CDT 8.1.0 as a baseline I was able to index the complete source using a peak of just over 1Gb of memory:
>>
>> CDT 8.1.0:
>> C/C++ Indexer: Project 'boost-1.55.0' (12 sources, 13,156 headers)
>>     Options: indexer='PDOMFastIndexer', parseAllFiles=true, unusedHeaders=useCPP, skipReferences=false,
>> skipImplicitReferences=false, skipTypeReferences=false, skipMacroReferences=false.
>>     Database: 461152256 bytes
>>     Timings: 1377747 total, 701404 parser, 157978 resolution, 215679 index update.
>>     Errors: 152 internal, 189 include, 154 scanner, 4261 syntax errors.
>>     Names: 1641458 declarations, 9460002 references, 912516(7.60%) unresolved.
>>     Cache[64MB]: 855744592 hits, 19246137(2.20%) misses.
>> Indexer: completed PDOMRebuildTask[1377840ms]
>>
>> Using the current master (at commit 23c2fd) I ran out of memory and crashed during indexing.  I raised my max memory
>> settings to 2Gb and was able to index this project using a peak of just under 1.5Gb:
>>
>> CDT master (commit: 23c2fd):
>> C/C++ Indexer: Project 'boost-1.55.0' (13 sources, 12,796 headers)
>>     Options: indexer='PDOMFastIndexer', parseAllFiles=true, unusedHeaders=useCPP, skipReferences=false,
>> skipImplicitReferences=false, skipTypeReferences=false, skipMacroReferences=false.
>>     Database: 496119808 bytes
>>     Timings: 3200221 total, 906062 parser, 639984 resolution, 220217 index update.
>>     Errors: 22 internal, 184 include, 195 scanner, 4625 syntax errors.
>>     Names: 1593995 declarations, 9480500 references, 882830(7.38%) unresolved.
>>     Cache[64MB]: 1832837748 hits, 201870349(9.92%) misses.
>> Indexer: completed PDOMUpdateTask[3200948ms]
>>
>> A few interesting things in this comparison:
>>
>> - 8.1.0 indexed more headers (but fewer sources)
>> - master has fewer unresolved symbols (although it found fewer declarations, so I'm not sure if that is an improvement)
>> - master finds more references
>> - 8.1.0 was more than *twice* as fast as master
>>
>> I've attached the jconsole heap usage graphs for these two cases.
>>
>> I don't have peak memory use for 8.2.0 (since I wasn't using jconsole yet when I did that test).  This release found
>> more headers, but is otherwise sort of between the previous 8.1.0 and master:
>>
>> CDT 8.2.0:
>> C/C++ Indexer: Project 'boost-1.55.0' (13 sources, 13,393 headers)
>>     Options: indexer='PDOMFastIndexer', parseAllFiles=true, unusedHeaders=useCPP, skipReferences=false,
>> skipImplicitReferences=false, skipTypeReferences=false, skipMacroReferences=false.
>>     Database: 477392896 bytes
>>     Timings: 2358690 total, 1244299 parser, 263195 resolution, 310024 index update.
>>     Errors: 12 internal, 184 include, 168 scanner, 4302 syntax errors.
>>     Names: 1666695 declarations, 9633150 references, 901119(7.39%) unresolved.
>>     Cache[64MB]: 724158478 hits, 19846710(2.67%) misses.
>> Indexer: completed PDOMRebuildTask[2359837ms]
>>
>> -Andrew
>>
>>
>> _______________________________________________
>> cdt-dev mailing list
>> cdt-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/cdt-dev
> 
> 
> 
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>