Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects
Hi Kevin et all,

Unfortunately IBM HeapAnalyzer solution doesn't work either. Reason is that it use java.util.ArrayList to store objectID, the size limit of ArrayList is Integer.MAX_VALUE since it's backed by an ordinary array which is 2,147,483,647.

Stack trace:
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyof(Unknown Source)
at java.util.ArrayList.grow(Unknown Source)
at java.util.ArrayList.ensureExplicitCapacity(Unknown Source)
at java.util.ArrayList.ensureCapacityInternal(Unknown Source)
at java.util.ArrayList.add(Unknown Source)
at com.ibm.jinwoo.heap.HprofTask.readHeapDump(HprofTask.java:1613)
...
at java.lang.Thread.run(unknown source)

com.ibm.jinwoo.heap.HprofTask.readHeapDump(HprofTask.java:1613):
          
          if (pass == 1) {
           
            this.objectCount++;
            this.objectList.add(new Long(objectID));
          } 

Thanks,
Roy

On Mon, Sep 23, 2019 at 4:47 PM Roy Zhang <roy.sunny.zhang007@xxxxxxxxx> wrote:
Thanks Kevin, I can download ha456.jar using ur link.
Now I am opening 91G heap via ha456.jar, hope I can open it.

Thanks,
Roy

On Mon, Sep 23, 2019 at 11:43 AM Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote:
Hi Roy, Download link works for me: http://public.dhe.ibm.com/software/websphere/appserv/support/tools/HeapAnalyzer/ha456.jar

--
Kevin Grigorenko
IBM WebSphere Application Server SWAT

Roy Zhang --- [EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects ---

From:"Roy Zhang" <roy.sunny.zhang007@xxxxxxxxx>
To:"Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date:Sun, Sep 22, 2019 8:28 PM
Subject:[EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects

Thanks Kevin for ur kindly advice! Unfortunately download link in that page doesn't work.
I happened to find a github repo (https://github.com/aragozin/jvm-tools/tree/master/hprof-heap) which can analyze heap dump via java API with some limitations (Only forward reference traversing), let me give it a shot.

Thanks,
Roy

On Fri, Sep 20, 2019 at 9:55 PM Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote:
Hi Roy,
Before IBM switched its focus to MAT, an IBMer built another (free) heapdump analyzer tool which you can also try: https://www.ibm.com/developerworks/community/groups/service/html/communityview?lang=ja&communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091
If it works, be careful of its "size" calculations (the number in brackets to the left of each object) and its leak suspects report because as far as I know they're built by heuristics more than a formal digraph and retained heap analysis.

--
Kevin Grigorenko
IBM WebSphere Application Server SWAT




From:        Roy Zhang <roy.sunny.zhang007@xxxxxxxxx>
To:        Memory Analyzer Dev list <mat-dev@xxxxxxxxxxx>
Date:        09/20/2019 02:28 AM
Subject:        [EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects
Sent by:        mat-dev-bounces@xxxxxxxxxxx



Thanks Andrew for ur quick feedback! 
I tried jprofiler and yourkit profiler, it throws the same error message as Eclipse MAT.
Regarding netbean profiler, it throws an strange message "an attempt was made to move the file pointer before the beginning of the file", I don't know the reason.
I happen to find one expert can open 150G heap (http://blog.ragozin.info/2015/02/programatic-heapdump-analysis.html, github: https://github.com/aragozin/jvm-tools/tree/master/hprof-heap), limitation is Only forward reference traversing, I will give it a shot.

>>>How many people hit the 2^31 object limit? Would 2^32 objects be enough or does it need to be more?
For my case, 91G heap have 3 billion object (shown by Yourkit profiler), 2^32 objects is enough for me. 

>>>Raising the MAT object limit sounds a big project - my first guess before even designing an approach would be 3 months of developers time, and unfortunately my day job is not working on MAT
I am interested in fixing it. If you could provide details, I am glad to contribute my effort to MAT :)

Thanks,
Roy

On Fri, Sep 20, 2019 at 3:49 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx> wrote:
Roy,

We have not done any work on increasing the limit on the number of objects MAT can handle. We do not have a workaround either. I don't know what the limits are for other memory analysis programs.

It might be possible to write a utility which reads a HPROF file, discards some objects and writes the remainder to a new file, but the result would be misleading unless there was some clever processing to decide  which objects to discard. There is HPROF reading and writing code in the org.eclipse.mat.hprof plugin which might help writing such a utility.

Raising the MAT object limit sounds a big project - my first guess before even designing an approach would be 3 months of developers time, and unfortunately my day job is not working on MAT. When I've finished on integrating parallel parsing improvements (bug 272324)  I'll think about it some more, but I don't see myself having the time to implement it. How many people hit the 2^31 object limit? Would 2^32 objects be enough or does it need to be more?

Andrew

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev


_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev

_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev

Back to the top