[news.eclipse.tptp] Some personal thoughts about profiling with TPTP

[Mauro Molinari <mauro.molinari@xxxxxxxxxxxx>]

Hello all.

After spending about an entire week trying to profile a medium-size 
application (just its starting phase, with data loading from a database) 
with TPTP, I would like to share with you my personal experience in 
order to understand your point of view.
Why?
Because I'm sorry, but I must admit that NetBeans profiler is many steps 
ahead TPTP... I don't like NetBeans and I think it's some years behind 
Eclipse in many areas, but there's no story about the profiler.

A co-worker of mine and I spent almost an entire week trying to make 
TPTP work as we wanted, without succeeding, while we were able to do the 
same thing with NetBeans profiler within one hour (!!!), even if we had 
to install it and use for the very first time!

I'm explaining why.

PRELIMINARY REMARK: in this post, when I talk about "profiling" I'm 
referring to Execution Time Analysis with JRE 1.5 or newer (actually, we 
used 1.5), with the following options:
- collect method CPU time information UNCHECKED
- Show execution flow graphical details SELECTED

Apart from the fact I couldn't make TPTP work on my own PC (but I thank 
Igor Alelekov very much for his support on this issue, see bug 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=201474), my collegue and I 
could make it work on a Linux system (equipped with an Athlon 64 2800+, 
1.5GB RAM) and on another Windows XP system (equipped with a Sempron 
2600+, 1.5 GB RAM).

First of all, under Linux there are some annoying issues:
- we couldn't use the IAC: we must use the stand-alone agent controller; 
it's not a big problem, although it makes TPTP not so "plug 'n' play" 
for the beginner
- even if we were using a powerful system, we had to calibrate the 
filters very strictly in order to get data in a reasonable amount of 
time and without using too much system resources; this required quite a 
big amount of time, because we had to exclude almost ALL classes and 
then include just those we wanted to analyse... the problem was to 
identify those classes and understand the difference between an 
inclusion filter like "com.something.*" and "com.something*"... However, 
the performance was quite poor even if we fine tuned the filters so 
carefully...
- on this system (running an Debian 4.1 with Gnome window manager), 
navigating through the Call Tree view is a pain... Just clicking on some 
of the entries (in order to highlight them) takes AGES... don't ask me why!
- we couldn't export the data: when we tried to do that, Eclipse froze, 
no progress bar appeared, the CPU went to 100% and, although we left the 
PC for several minutes, when we came back it was still there... so we 
had to kill Eclipse...
- we had alternate luck on profiling: the overall impression was that 
TPTP is not so stable and reliable, so we concluded that it was safer to 
close Eclipse and start it again from one profiling session to another

On the other system, we had better luck. Under Windows XP, in fact, the 
user experience seems a bit better:
- the IAC works and we used it
- there are no navigation problems in the Call Tree (the "pauses" shown 
when clicking on some entries were acceptable)
However, there are still many problems. First of all, the performance: 
we had to wait several minutes to complete the profiling of the load 
process of our application (even if the filters were fine-tuned) and it 
required more than 600 MB of RAM just to collect data (!!!).

Anyway, we wanted to do these simple tasks:
1) profile the application as it were and collect data
2) change the application startup algorithm
3) profile the application again and collect the new data
4) with both profiling data in the workbench, compare the results

We couldn't manage to perform these tasks, because:
- even if we could save data under Windows XP (honestly, I didn't try to 
"export", as on Linux, but simply to "Save" data in the workbench), 
after restarting Eclipse (it took AGES to start up again, I think it was 
trying to load the profiling data just saved...) I tried to navigate 
through that data but the views were... EMPTY!
- we coulnd't perform both profile sessions without restarting Eclipse, 
because TPTP requires too much memory: on that system, with 1.5 GB RAM, 
I couldn't start Eclipse with more the 1GB of maximum Java heap space, 
and it was not enough, because each single profiling session ate 
hundreds of MB of RAM (after manually starting the garbage collector 
from the Eclipse heap status bar, 600 MB were still allocated by 
Eclipse, even if the first profiling session was terminated)

Moreover:
- sometimes filters are not honoured: data collection seemed to stop at 
a certain point and execution time information about some classes and 
methods I would expect to find were not shown in the statistics; I had 
to remove and add again some filtering rules from one profiling session 
to another to make it more or less work as intended
- filters are not saved immediately: if Eclipse crashes and you restart 
it, the filters you added in the previous session are not saved; you 
must close Eclipse in a safe way in order to make them persistent from 
one restart to another
- if I have this object hierarchy:
class A{ public void a(); }
class B extends A{ public void b(); }
if I'm interested in profiling just a() method, when executed on a B 
instance, I have to include also B (not only A) in the filtering 
criteria, otherwise data is not collected... Maybe this is the wanted 
behaviour, but I found it hard (or, better: time consuming!!) to get to 
this conclusion
- in the call tree, I just see method names, without the corresponding 
classes: suppose you see a next() entry, it would be "nice" to know that 
you are talking about SomeClass.next() intead of, say, 
AnotherClass.next()... You have to right click and choose "Open Source" 
(wouldn't "Go To Source" be better? ;-)) to go to the corresponding 
source code
- we also would find it very useful if it were possible to define a 
profiling criteria like this: "start to profile the mathing classes but 
only after A.a() is invoked for the first time, and stop when B.b() is 
invoked".

I must admit that the application we tried to profile was not so small 
and loads a lot of data... but it is a real world application, were we 
need a profiling tool to understand how to make it better, if needed! We 
had to find out the bottle neck in the starting process and eliminate it...

At the end, we tried NetBeans profiler: using the Linux system, with 
less than 300 MB of RAM used by NetBeans, we could do 4 subsequent 
profiling sessions (!!!), filtering out JUST Java and Sun classes (so: 
no fine-tuning of filters were needed here to get a decent performance 
result!), navigate through data as if we were navigating through source 
code (I mean: the software was extremely responsive, it didn't give the 
impression of moving through some gigabytes of data or such...), save 
data in a couple of seconds and show graphs and statistics updated even 
in REAL-TIME!!! And without the need of any external agent controller 
process...

Maybe I wouldn't have even written here if I didn't experience such an 
impressive difference in performance and scalability of NetBeans 
profiler against TPTP, because I'm not the typical user that thinks it's 
funny to feed stupid wars like NetBeans vs Eclipse: however, I just 
wonder if there are some other reasons behind these results...

Does TPTP do also other things that NetBeans doesn't, when profiling the 
execution time analysis of Java code?
Is NetBeans using some "hidden" APIs or particular features of Sun VMs 
that Eclipse can't or doesn't use to profile applications?

Thanks for reading up to here and sorry for such a long post...

Mauro.