The Memory Analyzer tool will be presented for the third year in a row at the JavaOne conference. In the technical session TS-4118 Practical Lessons in Memory Analysis the speakers - Krum Tsvetkov (me) and Andrew Johnson - will show how many of the common memory-related problems can be solved using heap dumps and the Memory Analyzer tool. We are going to show many demos, using real-life heap dumps in different format - HPROF and IBM dumps. Based on these examples we will try to give some hints which we found useful in our troubleshooting experience, and which we think can be easily applied in practice.

The day after the session we will be available at the Eclipse booth (Thursday, June 4, 12:00 - 14:00) in the pavilion area. If you happen to be at the conference and have some questions, critics, comments for us, we will be happy to meet you there and get your feedback.

I hope to see many of you at the conference!

Eclipse has a Plug-in Registry view, which shows the state of all plug-ins in the running instance of Eclipse. But what if you need to analyze a crushed application to see what was deployed on it? Or, like in my case, need to understand a certain bundle dependency in RCP-application that has no Plug-in Registry view?

There was nothing specified in the manifest, but yet the bundle was able to load the necessary class. Later I found out that there was a bundle fragment that injected the dependency. Analysis of the heap dump didn’t help much as there wasn’t an easy query to visualize the bundles. But it did disclose the possibility to re-create the state of an OSGi framework from the heap dump. Some investigation and voilá:

Equinox Bundle Explorer may be open via Query Browser –> Eclipse –> Equinox Bundle Explorer

During the query execution the state of the OSGi framework is read from the heap dump. The query checks a bundle registry, a service registry and an extension registry and puts all the findings together in one resulting view:

One starts with bundles and then drills down into the bundle dependencies, extensions and services. The state of the bundle (active, resolved, starting, stopping, installed, uninstalled) is immediately visible to the right of the bundle name.
Depending on the bundle, it might have:

• dependencies and dependents
• services contributed and consumed
• extension points and extensions
• hosted bundle fragments (A bundle fragment is an “incomplete” bundle. In order to function it needs to be attached to the bundle host. When attached, the fragment injects resources to the host bundle. These resources are loaded by the host class loader.)

A plug-in is an OSGi bundle. On top of OSGi standard it can register extension points and be extended with new functionality by other plug-ins. In Extension Points folder you can see which extension points a plug-in defines and which bundles declare extensions for it:

Each extension point has a name and an XML schema, which specifies the structure of the meta-data that extensions must comply. Extension contains information on the attributes and values, defined in the XML schema of the extension point:

An OSGi service is a Java object, registered into a framework with a set of properties and published under the name of a Java interface:

A bundle might register several instances of one service, but those instances should have different properties.
You can browse Bundle Explorer by services (via the toolbar menu action). Each service lists its properties, a bundle, that has registered it, and bundles using it:

Browse by extension points:

Using a context menu on a bundle, service, extension point, etc. you can jump directly to the implementation object, we’ve read the information from.

Open issues

Re-creation of the state of OSGi framework from the heap dump is dependent on internal implementation classes. And because they are internal, they can change without giving a notice. If you observe abnormalities, please let me know. One thing I’ve already noticed is that information about the extension points and corresponding extensions could not be retrieved from the heap dumps, acquired in debug mode.
Equinox Bundle Explorer is in beta state. Your feedback, suggestions for improvements and feature requests are welcome!

Just a quick note: if you are in or around Mannheim, Germany, on November 25th, you might want to join Markus Kohler’s presentation of the Memory Analyzer at the Java User Group Mannheim. The talks is at the University Mannheim at 7pm.

There was one question which popped-up at every Memory Analyzer demo we did in the last couple of years – “Is the Memory Analyzer able to read heap dumps also from IBM VMs”? And we always had to give negative answers. Not any longer!
After some joint efforts it is now possible to analyze memory problems which occurred on IBM VMs with the help of IBM’s DFTJ technology and the Memory Analyzer tool.

Overview

The Diagnostic Tool Framework for Java (DTFJ) is a Java API from IBM used to support the building of Java diagnostics tools. Using one and the same API – DTFJ - tools can read data from different data sources in a unified way. The IBM DTFJ adapter enables MAT to work with system dumps from IBM Virtual Machines for Java version 6, version 5.0 and version 1.4.2.

Setting-up the Tools

In order to analyze IBM system dumps with Memory Analyzer, one needs to install the DTFJ adapter into the Memory Analyzer. The DTFJ adapter as well as installation instructions are available here.

Getting Data for Analysis

A detailed documentation on using DTFJ with MAT is added to the “Help” of MAT once the DTJF adapter is installed. There it is described how dumps can be obtained and pre-processed. Further reading about DTFJ can be found in the “Diagnosis documentation”

Here I will only roughly present the process:

The minimum-required versions of the IBM JDKs are:

Analyzing the DTFJ Extracted Dumps

The analysis with the Memory Analyzer remains the same as for HPROF dumps. This means that if you have already some experience with the tool, you can just continue to do the analysis in the same way did before also for IBM dumps.

If you are new to the tool and need some “first steps” help, here are some resources which could help:

If you want to see the Memory Analyzer & DTFJ in action, you can visit the “New & Noteworthy” short session at Eclipse Summit Europe 2008.

The Memory Analyzer has powerful views to analyze a Java heap dump. But what if you look for those odd objects? Here the build-in Object Query Language (OQL) comes to the rescue. This blog introduces the basic syntax, commonly used queries and practical hints that help you dissect and analyze your objects.

The object query language (OQL) is very similar to SQL. Just think of classes as tables, objects as rows and attributes as columns. So, let’s open the OQL view (4th button from the left) and start with a very simple query (No Dump?):

SELECT * FROM java.lang.String

Alternatively, we can select the fields to be displayed:

SELECT toString(s), s.count, s.value
FROM java.lang.String s

Read the rest of this entry »

There is a common understanding that a single snapshot of the Java heap is not enough for finding a memory leak. The usual approach is to search for a monotonous increase of the number of objects of some class by “online” profiling/monitoring or by comparing a series of snapshots made over time. However, such a “live” monitoring is not always possible, and is especially difficult to be performed in productive systems because of the performance costs of using a profiler, and because of the fact that some leaks show themselves only rarely, when certain conditions have appeared.

In this blog will try to show that analysis based on a single heap dump can also be an extremely powerful means of finding memory leaks. I will give some tips how to obtain data suitable for the analysis. I will then describe how to use the automated analysis features of the Memory Analyzer tool, which was contributed several months ago to Eclipse. Automating the analysis greatly reduces the complexity of finding memory problems, and enables even non-experts to handle memory-related issues. All you need to do is provide a good heap dump, and click once to trigger the analysis. The Memory Analyzer will create for you a report with the leak suspects. What this report contains, and how the reported leak suspects are found is described below.
Read the rest of this entry »

Markus Kohler blogged about Analyzing Memory Consumption of Eclipse:

During my talk on May 7 at the Java User Group Karlsruhe about the Eclipse Memory Analyzer I used the latest build of Eclipse 3.4 to show live, that there’s room for improvement regarding the memory consumption of Eclipse.

He goes on to have a closer look at the spell checker and looks at duplicate strings.

Now you may think, that this guy is bashing Eclipse, but that’s really not the case.
If you beg enough, I might also take a closer look at Netbeans :]

If nothing else, it shows how relative simple it is to gain some insights about the memory of your application…

The Hotspot Virtual Machine employs generational garbage collection: a young generation holds recently created objects, the tenured generation those objects which survived (multiple) major Garbage Collections (GCs). The heap dump contains all objects from both spaces, even though we cannot tell anymore to which generation they belonged.

The permanent generation (or: perm space) is a different beast: It is used to store class and method data as well as interned strings. Just like heap space, you can also run out of perm space. That’s what happens for example if you install too many plug-ins to your Eclipse IDE: OutOfMemoryError: PermGen full. In this case, you have to increase the available perm space using -XX:MaxPermSize.

Too many plug-ins are only one reason: One can run out of perm space due to too many interned Strings or because of leaking class loaders. Increasing perm space will only help for so long.

Because a heap dump does not really contain a lot of information about perm space, perm problems are difficult to tackle. Recently, I found this great article by Sporar, Sundararajan and Kieviet. The authors shed some light on the permanent generation. Of course, I had to check right away if and how I can use the Eclipse Memory Analyzer to analyze this “unknown” generation. This is what this blog is about.
Read the rest of this entry »

Hello,

I just wanted to take the opportunity to thank everybody who joined our technical session at Java One 2008 - “Automated Heap Dump Analysis for Developers, Testers, and Support Employees”.
I was very happy to see more than 400 people in the room, despite of the fact that the session was in the afternoon of the very last day of the conference. Thank you! It was a real pleasure for us the speakers.

As a project that is relatively new at Eclipse (was recently contributed by SAP) we are very interested to get some feedback from you - both about our JavaOne session and about the tool.

So I hope that we can get some comments added to this blog from the people who saw the session @ Java One and want to spend several minutes to give us their valuable opinion.

And for any feedback, suggestions or comments on the tool itself, please use our newsgroups.

For anybody who is interested in the topic and in the Memory Analyzer tool, but didn’t have the chance to play with it yet - our project page is http://www.eclipse.org/mat/.

The goal of this article is not to provide you with a step-by-step approach in finding memory problems, but to reveal several powerful features of the Memory Analyzer, which are irreplaceable in memory leak hunting. These features might be unified into one topic – aggregation.

Memory Analyzer - is a powerful Java heap analyzer. It makes it possible to find a memory leak or reduce memory consumption by looking into the heap dump only. Our wiki page provides detailed information on how to acquire a heap dump.

Sometimes you open a heap dump in Memory Analyzer and see nothing … but a pile of small objects:

As overview pane shows, the majority of retained memory is more or less equally distributed among several objects, which makes it difficult to see a suspect for a memory leak at first sight.

Grouping in Dominator Tree

The pie in the overview pane is a graphical representation of the Dominator Tree, which you can open from the toolbar using the button

The Dominator Tree is built out of an object graph. Each object in the tree is responsible for keeping its children alive. The tree is sorted in descending order according to the size an object retains in memory.
Now, if we group the dominator tree by class (using the drop-down toolbar menu), we will see a different picture:

We can see that 700 objects of class com.foo.bar.menu.MenuEntry retain 46 % of the total heap space. Suddenly we see an unequal distribution of the retained memory.
How is it possible that we see it only now? The answer is that in this heap dump we do not have one single big object, but a group of leaking objects. Remember, the overview page of the heap dump only displays the distribution of retained memory heap on a per-object basis. So, when we group the dominator tree by class, we group our leaking objects to their origin class and thus get an accumulated retained heap.
Let’s find out who is referencing all those com.foo.bar.menu.MenuEntry objects. For this select List objects –> with incoming references from the context menu. As a result we will get an object reference graph:

If we drill down into the com.foo.bar.menu.MenuEntry, we can see the chain of objects referencing our suspect. Unfortunately, in this case it is not really helpful as we see only a LinkedList structure.

Grouping referrers to the class level

As we have seen looking at the incoming references graph for our suspect objects did not reveal a lot. Grouping referrers to class level might be more helpful. This operation is performed via a context menu. Back in the dominator tree grouped by class, right-click on our suspect com.foo.bar.menu.MenuEntry and select Show objects by class –> by incoming references:

Grouping referrers by class allows us to see a simple and easy to understand reference chain:

We can now see, that class com.foo.bar.menu.MenuEntry is referenced by a LinkedListEntry, which we already knew. But as we continue we see that this entry is again referenced by a LinkedListEntry (this is the head of the LinkedList) and by the LinkedList. LinkedList itself is referenced by two classes, one of which is of no interest as it is from the java.* package (which for now we assume to be unproblematic). The other one, however, is of interest.
The easiest way to find out what keeps our memory leaking group of objects alive in the heap is to check the path to the garbage collection roots. However, paths to garbage collection roots can only be found for a single object but we are interested in finding a common chain of references to a group of objects. Well, this is also possible.

Merging Paths to Garbage Collection Roots

The Merged Paths to GC Roots view shows the shortest paths from the GC roots to each instance of the selected class. You can open this heap editor using the context menu Merge Shortest Paths to GC Roots on the com.foo.bar.menu.MenuEntry class instance:

This operation explores all the paths from the different objects to their GC roots at once and finds a common chain of references for this group of objects. It is a common procedure to exclude weak and soft references from the result as they cannot be the main reason of keeping the objects of interest alive.

There we got the reason: one instance of the class com.foo.bar.menu.FoodSupplies keeps in memory 700 instances of the class com.foo.bar.menu.MenuEntry. Now is the right time to check what is going on in the code and fix the problem.

Conclusion

Apparently, Memory Analyzer has more aggregation features to offer. Some of them will be covered in our future blogs.