Re: [mat-dev] Parallel parsing

[Andrew Johnson <andrew_johnson@xxxxxxxxxx>]

I've been looking
at memory usage with parallel parsing. The dumps java_pid12784.0001.hprof
and java_pid21528.0001.hprof were generated using org.eclipse.mat.tests.CreatePerformanceDump.

Dump	java_pid12416.0001.hprof	java_pid12784.0001.hprof	java_pid21528.0001.hprof	importCamelInOxygen.hprof
Description	dump from IBM RTC Eclipse	100 String[2000000]	2 String[20000000]	dump from Eclipse
File length	1,427,397,269	1,834,724,749	2,136,737,980	6,719,753,673
Used heap dump	195.9 MB	915.3 MB	1.5 GB	3.9 GB
Number of objects	4,542,380	5,984,460	50,003,364	110,991,221
Number of classes	33,555	434	434	18,087
Baseline	-Xmx371m	-Xmx212m	-Xmx1496m	-Xmx3142m
Total time/ms	69939	122047	144111	474459
With parallel parse	-Xmx457m	-Xmx3261m	-Xmx1910m	-Xmx3221m
Total time/ms	33766	84993	103831	333506
Parallel parse with batch limits	-Xmx464m	-Xmx593m	-Xmx1850	-Xmx3376m
Total time/ms	31929	88913	113062	332825
Sequential stream parse with batch limits	-Xmx462m	-Xmx231m	-Xmx1533m	-Xmx3183
Total time/ms	41842	111086	132993	430719

These are approximate
timings on my i7-8850H 6 core, 12 threads machine.

Parallel parsing
does use more memory, particularly with java_pid12784.0001.hprof. I then
attempted to reduce the memory usage.

HProfParserHandlerImpl
   
    Null ids reference in parseHeapObject for object
arrays once they have been copied.
Pass2Parser
   
    In trySplit() if there are too many reference, end
the buffer early.
   
    readDumpSegments() - StreamSupport.stream. Try sequential
rather than parallel operation.
IntArray1NWriter
   
    Keep track of the size of ArrayList<SetTask>
and publishTasks() early if there are too many references.

These results
show that we can have the parallel parsing code changes without increasing
memory usage too much. However, we may need to consider an option to switch
from parallel to sequential streams. I think we will soon be ready to start
merging.

Andrew Johnson




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU