Community
Participate
Working Groups
Reported on the mailing list, a few projects that together contain a large bunch of source are running out of memory: === I have a problem using AspectJ within Eclipse (AJDT). Problem description: Following project dependencies. Project1 | Project2 / \ Project3 Project4 / | \ / Project5 | Project6 Project7 (aprox. 10K Classes, 10 Aspects) All Projects are AJ projects. The aspects are located in Project1 and 3 and should be woven into all projects that depend directly or indirectly on the containing project. All projects have a project dependency to the project(s) above. And also an entry in the aspect path to the bin folder of the project(s) above. Both dependencies are exported. I might be wrong but AFAIK this should be correct. Most of the pointcuts use execution() compared with within(). There are only few that user call(). The problem is that the build fails since it runs out of heap space and Eclipse crashes(Heap size is set to 3 GB). I've tried minmalModel=true and also typeDemotion=true. No changes. Environment Info: Eclipse 3.6.1 (Helios) WinXP Pro SP2 running as a VM 10 GB RAM Any ideas what could be wrong with this setup that causes this excessive use of heap? If some important information is missing please tell. ===
On 28 January 2011 06:04, Florian Weiss wrote: > Hi, > > Just did som more builds. One project at a time. > I have 1107 markers in 9754 files in total. > > I also analyzed the size of the model dump and it was interesting to see: > > Project2 and Project3 are rather big (50% of the files) the later projects are smaller. > So with the build of the later projects the dump just increased by a couple hundred lines (~500) but the heap usage increased by ~200 MB so it seams ther is no relation. Is that true? or weird? Is that with typeDemotion on or off? You could use a profiler or heap dump tool to see where all the space has gone, but 200M doesn't sound huge if you have a 3G heap, where is the rest of it being used...
> Is that with typeDemotion on or off? typeDemotion was off only minimalModel was on true. > You could use a profiler or heap dump tool to see where all the space has gone, > but 200M doesn't sound huge if you have a 3G heap, where is the rest of it > being used... it increased by 200 before it was at 2.9GB. It just seemed that the heap space increased way to much for such a tiny little project that was build.
A large increase in heap like that could be countered by typeDemotion. When using inpath with a project, we may need to pull in a lot of type information. TypeDemotion will ensure it is gotten rid of more actively. Anyway, I've committed the changes to further reduce the model. On my project I was checking the heap usage in eclipse (using the built in monitor as a crude measure). Without the minimalModel option I was using 281M after a full build, with the option on I was using 234M, with these extra changes I was using 226M. This is not a particularly huge project so you should see even more payoff. The size of the 'data' dumped in the ajdt event trace view after a build was, respectively, 7.5meg, 2.89meg and 2.16meg. The AJDT build containing these changes should be out later today.
Hi Andy, just tried the new build. It seems that the model is free of classes that shouldn't be there thats good. But when I did the build I still had the "Model Sanity Check" Log Filter enabled. And no i get alot of output from there. Not for Project1 but for all following projects. It says that: Java Element does not exist: =Project2/binaries<...(Handle of a around advice from an aspect of Project1).. It is the source relationship of ...(a relationship that is within Project2). As far as i can see the weaving works, i was just wondering why this output is generated? Florian
So, on the memory thing, are things behaving much better for you now? I am interested in any numbers if you have them, since your projects are much bigger than mine. > But when I did the build I still had the "Model > Sanity Check" Log Filter enabled. And no i get alot of output from there. > > Not for Project1 but for all following projects. It says that: Java Element > does not exist: =Project2/binaries<...(Handle of a around advice from an aspect > of Project1).. > It is the source relationship of ...(a relationship that is within Project2). I imagine this is because that sanity checking is for project builds that don't consume aspects from outside of the project - it hasn't been written to cope with the 'binary' handles we create that point to locations outside of the project. If it isn't causing you a problem, I would just ignore it, or you could raise an AJDT issue to make that checking smarter, but it won't be particularly high priority I don't think.
> So, on the memory thing, are things behaving much better for you now? I am > interested in any numbers if you have them, since your projects are much bigger > than mine. Build options: minimalModel on, typeDemotion off Dump size with old Version: 47K lines Dump size with new modified Version: 29K lines Heap space usage with new Version after build: 2.8 GB (still quite big, but i guess now only typeDemotion on will help) More numbers will follow... > I imagine this is because that sanity checking is for project builds that don't > consume aspects from outside of the project - it hasn't been written to cope > with the 'binary' handles we create that point to locations outside of the > project. If it isn't causing you a problem, I would just ignore it, or you > could raise an AJDT issue to make that checking smarter, but it won't be > particularly high priority I don't think. Ok. It is not an issue to me, I was just curious since I couldn't remember that i had this output with the old version. I mentioned the double dependency between the projects, is there a way to avoid this. Eclipse now knows all the classes twice and the initialization of the search index takes quite long? If i put the whole project on the aspect path it seems not to work since it also has the sources for the aspects. Is there already a bug adressing this problem? I couldn't find one. Florian
I'll have a think on the double project links setup. Do you see search indexing actually taking longer, or are you just thinking it will? Yes, typeDemotion is the way for you to recover the next big chunk of memory.
(In reply to comment #7) > I'll have a think on the double project links setup. Do you see search > indexing actually taking longer, or are you just thinking it will? It will, when i opened the "Open Type" dialog i had to wait for around 2 minutes. My suggestions would be: If a project is added to the aspectpath the aspect compile only looks in the bin folder of the project since. Or are there cases where you really need the whole project on the aspectpath? > Yes, typeDemotion is the way for you to recover the next big chunk of memory. Ok, good to know for sure. So now its either good performance and a lot of memory or a bad performance but low memory. Thanks for your efforts. Florian
I did another test with typeDemotion=true, and while i was waiting for the build to finish (17 minutes). I was wondering why the initial build time is affected by this Option. Andy wrote on his blog that it throws away woven types as soon as they are written on the disk. So the downside should be that the information has to be read from the disk again if needed. Are the woven types needed twice in the initial build so they have to be retreived from the disk again? or is the discard process so expensive? Florian PS: Sorry for the double post.
Yes, they may be needed again. If Foo is woven and then dumped to disk, sometime later we may encounter a subtype of Foo and maybe how we treat this subtype (the matching perhaps) depends on the shape of Foo, so we go back out to disk and pull it back in again. There is another bug open somewhere for this area - to improve the strategy such that types are anchored a little longer (since the usual case is that the next type you hit might reference the type you just finished with). If we held onto them for, say, the next 30/40 weaves, we could keep things around that are likely to be used, but throw those that are not. Or maybe track usage and discard those that haven't looked up in the last few weaves. What is the compile time without typeDemotion on? > My suggestions would be: If a project is added to the aspectpath the aspect > compile only looks in the bin folder of the project since. Or are there cases > where you really need the whole project on the aspectpath? Not quite sure I understand this suggestion. what is 'the aspect compile' - do you mean the compile of the project that has the aspectpath set? AspectJ will look through the aspectpath to discover aspects, it will use the aspectpath and inpath and classpath to resolve references from the project code and any aspects discovered on the aspectpath. Really the aspectpath exists because there is no easier way to tell the compiler more directly what the aspects are from some bundle of class files. We now support the use of aop.xml at compile time, which is a potential solution, although maybe we could have a different kind of dependency in Eclipse - less like a path - because it is straightforward for AJDT to ask the project on which you depend what aspects were in its build.
