Re: [mat-dev] Parallel parsing

[Andrew Johnson <andrew_johnson@xxxxxxxxxx>]

I have merged Jason
Koch's changes for parallel HPROF parsing - thank you Jason!
https://bugs.eclipse.org/bugs/show_bug.cgi?id=277422

I have then made
further updates to avoid API changes and to try to limit peak memory usage.
We will need to go up to version 1.10 before release as we now need Java
8 (and performance changes like this are significant, and there are hidden
API changes in that some APIs are now thread safe etc.)

It's now time
to start testing, to see if parsing is now faster, whether it uses more
memory and any other comments.

Some tuning constants:
IndexWriter.java
     
  //
length of set() queue to buffer before writing to output as a batch
     
  private
static
final
int
TASK_BUFFER_SIZE
= 1024;
     
  /**
Number of ints to write out in the buffer */
     
  private
static
final
int
TASK_BUFFER_MAX_OBJECTS
= 10000;
     
  /**
Size of all arrays in the buffer */
     
  private
static
final
int
TASK_BUFFER_MAX_MEMORY
= 20000;

Pass2Parser.java
     
  static
final
int
BATCH_SIZE
= 512;
     
  static
final
long
MAX_MEM
= 1000000;
Parallel parsing:
     
  Stream<HeapObject> heapObjects
= StreamSupport.stream(
     
                  new
HeapObjectParser(length),
true);

Andrew




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU