| Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects |
Thx Andrew for ur quick response, let me checkout MAT source code and have a check.Thanks,RoyOn Mon, Sep 30, 2019 at 4:48 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx> wrote:Roy,
As you saw in the discussion https://www.eclipse.org/lists/mat-dev/msg00324.html fixing the limitation is a big challenge.
Internally MAT uses int variables to represent objects in the dump and as indices into tables for object types, outbounds, inbounds etc. These values are often uses as indices into Java arrays, which have a 2^31-1 limit too.
MAT is extendable and has an API so this use of non-negative int values to represent objects is visible to extensions and changing this could be a breaking API change
Some ideas:
Create an annotation type for object ID.
Find out all the places where an int is used as an object ID.
Use the annotation to tag all the variables, fields, etc..which hold an object ID (For 1.10 we will move up to Java 8 which allows annotations in more places.)
Now if we allowed negative values for object ID (expanding the limit to 2^32 - 1?) or made it a long it would be easier to see where problems could occur.
I think there are so many assumptions in the code that object ID is a non-negative integer that extending it to negative integers too would be a big job. The index code would need careful thought. If you made a list of all the places where object ID was used and assumed to be positive then that would be a useful first step.
Enhancing MAT in this way won't be quick - the parallel parsing changes are next on my list and with my limited time available they are taking longer than I would like.
Andrew
From: Roy Zhang <roy.sunny.zhang007@xxxxxxxxx>
To: Memory Analyzer Dev list <mat-dev@xxxxxxxxxxx>
Date: 30/09/2019 05:12
Subject: Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects
Sent by: mat-dev-bounces@xxxxxxxxxxx
Dear Andrew and MAT experts,
https://github.com/aragozin/jvm-tools/tree/master/hprof-heap can't help me, and https://heaphero.io/ (most famous commercial tool ) can't help me, either.
I do see that there is opportunity/benefit for us to optimize MAT for this limitation. I would be glad to provide my contribution for MAT. Could u please kindly provide details for this limitation and I can begin to work on it.
Really appreciate ur great help in advance!
Thanks,
Roy
On Mon, Sep 23, 2019 at 7:18 PM Roy Zhang <roy.sunny.zhang007@xxxxxxxxx> wrote:
Hi Kevin et all,
Unfortunately IBM HeapAnalyzer solution doesn't work either. Reason is that it use java.util.ArrayList to store objectID, the size limit of ArrayList is Integer.MAX_VALUE since it's backed by an ordinary array which is 2,147,483,647.
Stack trace:
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyof(Unknown Source)
at java.util.ArrayList.grow(Unknown Source)
at java.util.ArrayList.ensureExplicitCapacity(Unknown Source)
at java.util.ArrayList.ensureCapacityInternal(Unknown Source)
at java.util.ArrayList.add(Unknown Source)
at com.ibm.jinwoo.heap.HprofTask.readHeapDump(HprofTask.java:1613)
...
at java.lang.Thread.run(unknown source)
com.ibm.jinwoo.heap.HprofTask.readHeapDump(HprofTask.java:1613):
if (pass == 1) {
this.objectCount++;
this.objectList.add(new Long(objectID));
}
Thanks,
Roy
On Mon, Sep 23, 2019 at 4:47 PM Roy Zhang <roy.sunny.zhang007@xxxxxxxxx> wrote:
Thanks Kevin, I can download ha456.jar using ur link.
Now I am opening 91G heap via ha456.jar, hope I can open it.
Thanks,
Roy
On Mon, Sep 23, 2019 at 11:43 AM Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote:
Hi Roy, Download link works for me: http://public.dhe.ibm.com/software/websphere/appserv/support/tools/HeapAnalyzer/ha456.jar
--
Kevin Grigorenko
IBM WebSphere Application Server SWAT
Roy Zhang --- [EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects ---
From: "Roy Zhang" <roy.sunny.zhang007@xxxxxxxxx> To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx> Date: Sun, Sep 22, 2019 8:28 PM Subject: [EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects
Thanks Kevin for ur kindly advice! Unfortunately download link in that page doesn't work.
I happened to find a github repo (https://github.com/aragozin/jvm-tools/tree/master/hprof-heap) which can analyze heap dump via java API with some limitations (Only forward reference traversing), let me give it a shot.
Thanks,
Roy
On Fri, Sep 20, 2019 at 9:55 PM Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote:
Hi Roy,
Before IBM switched its focus to MAT, an IBMer built another (free) heapdump analyzer tool which you can also try: https://www.ibm.com/developerworks/community/groups/service/html/communityview?lang=ja&communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091
If it works, be careful of its "size" calculations (the number in brackets to the left of each object) and its leak suspects report because as far as I know they're built by heuristics more than a formal digraph and retained heap analysis.
--
Kevin Grigorenko
IBM WebSphere Application Server SWAT
From: Roy Zhang <roy.sunny.zhang007@xxxxxxxxx>
To: Memory Analyzer Dev list <mat-dev@xxxxxxxxxxx>
Date: 09/20/2019 02:28 AM
Subject: [EXTERNAL] Re: [mat-dev] How can we open huge heap dump having more than 2, 147, 483, 639 objects
Sent by: mat-dev-bounces@xxxxxxxxxxx
Thanks Andrew for ur quick feedback!
I tried jprofiler and yourkit profiler, it throws the same error message as Eclipse MAT.
Regarding netbean profiler, it throws an strange message "an attempt was made to move the file pointer before the beginning of the file", I don't know the reason.
I happen to find one expert can open 150G heap (http://blog.ragozin.info/2015/02/programatic-heapdump-analysis.html, github: https://github.com/aragozin/jvm-tools/tree/master/hprof-heap), limitation is Only forward reference traversing, I will give it a shot.
>>>How many people hit the 2^31 object limit? Would 2^32 objects be enough or does it need to be more?
For my case, 91G heap have 3 billion object (shown by Yourkit profiler), 2^32 objects is enough for me.
>>>Raising the MAT object limit sounds a big project - my first guess before even designing an approach would be 3 months of developers time, and unfortunately my day job is not working on MAT
I am interested in fixing it. If you could provide details, I am glad to contribute my effort to MAT :)
Thanks,
Roy
On Fri, Sep 20, 2019 at 3:49 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx> wrote:
Roy,
We have not done any work on increasing the limit on the number of objects MAT can handle. We do not have a workaround either. I don't know what the limits are for other memory analysis programs.
It might be possible to write a utility which reads a HPROF file, discards some objects and writes the remainder to a new file, but the result would be misleading unless there was some clever processing to decide which objects to discard. There is HPROF reading and writing code in the org.eclipse.mat.hprof plugin which might help writing such a utility.
Raising the MAT object limit sounds a big project - my first guess before even designing an approach would be 3 months of developers time, and unfortunately my day job is not working on MAT. When I've finished on integrating parallel parsing improvements (bug 272324) I'll think about it some more, but I don't see myself having the time to implement it. How many people hit the 2^31 object limit? Would 2^32 objects be enough or does it need to be more?
Andrew
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev