Adrian,
What do the structure model numbers look like for 3a? It looks to me like
the times for the cvs head version are increasing more than linearly in the
number of aspects added. The 1.1.0 final numbers look linear in the number of
aspects (i.e., proportionate to the total # of relations).
Ron Bodkin
Chief Technology Officer
New Aspects of Security
m: (415) 509-2895
------------Original Message-------------
From: "Adrian Colyer" <adrian_colyer@xxxxxxxxxx>
To: aspectj-dev@xxxxxxxxxxx
Cc: ron.difrango@xxxxxxxxxxxxxx
Date: Tue, Sep-9-2003 3:24 AM
Subject: [aspectj-dev] ajc scalability (structure model)
I've been running some scalability
tests against the ajc compiler and structure model - motivated by Ron
DiFrango's performance problems with larger projects in AJDT and also by our
recent problems with the structure model in the head. This is a relatively
long note... but please keep reading to the end as it tells us a lot about the
scalability of our structure model generation in 1.1.1.
First some background. All numbers are meaningful only
on my machine - what's interesting here is the relative scalability, not the
absolute numbers. The project source files are synthetic - each project
consists of n classes, named C_1..n, each with 10 methods. The method bodies
contain a single System.out.println() - not representative of the compilation
workload on a real project, but enough for these purposes. Second a
warning - these numbers are all based on the ajc code in CVS head on 8th
September, they cannot be used to draw any conclusions about the 1.1,0 release
currently in the field (see last table for one exception to this).
Use a fixed width font for viewing the
numbers that follow...
Base
comparison, ajc vs javac (all times in seconds)
Project Size javac(1)
ajc 10
2 2 100
3 4 500 6
8
1000 12 15
2000
22 24 4000 67
55(2)
(1) Sun JDK
1.4.1 (2) This shows ajc starting to
perform better than javac... on 'real' large projects (where the method bodies
cause the compiler to do more work) we see that ajc is about 10% faster than
javac.
Compilation time grows
linearly with project size - good.
Base comparison, structure model generation
Project Size ajc
ajc -emacssym 10
2
2 100
4 4 500
8 12 1000 15
24 2000
24 55
4000
55 108
Compilation time still growing linearly with project size -
good.
This shows that the compile
time is roughly doubling with the -emacssym option specified. One possible
explanation here is that in this case the compilation time is bounded by i/o.
With the -emacssym option specified we write twice as many files to disk (one
.ajesym per class). This extra file writing is not done when generating
structure models for IDEs, so don't get too worried by it here, I'm simply
using -emacssym as a convenient way of forcing structure model generation on
the command-line, and it gives us a baseline for what follows...
Cost of adding aspects to the project -
straight compile
Project Size
ajc ajc +1a(1) ajc +2a(2) 10
2 3
3 100
4 7
7 500
8 15
16
1000 15
26 27
2000 24
48 51
4000 55
98 104
(1) Added a BeforeAll aspect that adds advice before all method
executions (10 methods in each class). (2) Added an AfterAll aspect that adds advice after returning from all
method executions
These are very
pervasive aspects. A hypothesis from these figures is that the compile
time is proportional to the number of joinpoints to be woven (not the number
of aspects), and stays within 150% of the no weaving (no aspects) compilation
time. On 'real' projects adding a variety of aspects (some very pervasive,
some less so) we see between 30% and 50% increase in compile time, which backs
this up.
Cost of adding aspects to
the project - with structure model generation
Project Size ajc
ajc +1a ajc +2a -emacssym
-emacssym -emacssym
10 2
3 3 100 4
7 8
500
12 21 23
1000
24 74 145
2000
55 142 286
4000
108 355 779
Oh dear. This is clearly no longer linear
with respect to the number of files in the project. The numbers suggest that
the dominant factor is the creation of a relationship in the structure model -
see how the numbers roughly double from the +1a to the +2a column. A
1000 file project with the 2 aspects added creates 20,000 relationships, and
takes about the same time as a 2000 file project with one aspect, which also
creates 20,000 relationships.
I
thought it would be interesting to run the same tests against 1.1.0 (the old
structure model generation code) to see how that fared...
Cost of adding aspects to the project - with structure
model generation, ajc v.1.1.0
Project Size ajc +1a ajc +2a
-emacssym -emacssym 1000 39
45 2000
89 112
4000
227 327
This is much better than 1.1.1, but still non-linear. (I also measured compile times without the -emacssym
option, which as expected are unchanged across 1.1.0 and 1.1.1).
These numbers were all taken from CVS
head. There are at least two known problems in the head, relating to changes
made last Thursday / Friday: 1) the
cache that Andy & I added is unfortunately missing (though I thought your
email said you'd left it in Mik? - missing a few changes in the
commit??) 2) *serious*: the changes
made to the path for source files broke all the ide support (structure node
could never be found for any file, so AJDT et al. could not display
anything).
Andy, George and I are
investigating and will restore the cache and IDE function, then re-run the
numbers. It seems mad that the cost of creating a relationship should be
dwarfing the cost of compilation and weaving, which intuitively are much more
expensive tasks! Andy has also exposed the option (which Mik added - thanks)
to turn off structure model generation in AJDT. This brings AspectJ
compilation times back to or better than Java compiles within Eclipse - but at
the cost of losing all outline and relationship information.
--
Adrian Adrian_Colyer@xxxxxxxxxx
|