Community
Participate
Working Groups
I would like to be able to create a state.dat within my own builder which the java builder understands. The reasons/explanation behind this is: My particular situation is that I'm working on the development of AJDT and want to allow users to have two projects, A and B, where A depends on B, project B is an AspectJ project (so contains aspects as well as java classes) and project A is a normal java project which has a project dependency on project B (via Project > properties > project references). What I would like the JavaBuilder to do when it builds project A is understand project B. Unfortunately, because project B has an AspectJ builder and not a Java one it doesn't have the build state which the java builder is looking for. Consequently, java project A doesn't build and has the "please rebuild prerequisite project A first" type message. Rebuilding project B does nothing to help the situation because it doesn't create a valid state file. (this is all buried in the call to isWorthBuilding() in the JavaBuilder) We have got around this by forcing a java build first on creation of new AspectJ projects which creates the state file and keeps the java builder happy. However, there is the one remaining case on a clean checkout of an AJ project from CVS when the creation of the AJ project doesn't go through this route. An obvious answer is to force it to do this, however, a nicer answer would be for either the javabuilder to understand that our project is an AJ project and that it can still build, or for us in our AspectJ builder to be able to create a state which the javabuilder understands. We have tried switching the dependencies to be class folder ones, and in this case everything builds. Unfortunately, not only are there a lot of special cases with class folder dependencies (which are dealt with automatically with project depenendencies i.e. to do with exporting jars), but also features like finding references in the workspace don't work. Therefore, keeping the project dependencies (or the dependencies how the user defines them) is the nicest (and cleanest) solution (switching the dependency was advised in bug 56703) It would be really nice if in the builder within AJDT, we can create a meaningful state.dat which the java builder understands. That way we could take advantage of the other nice things the state keeps track of (and not just use it to keep the java builder happy with project dependencies). Thanks, Helen
Kent - couldn't we simply remove the prereq build state check in the case where the prereq project doesn't use the Java builder ? i.e. if one disables the Java builder, we always trust it to be correctly build (if not, this isn't our problem any longer).
Let's treat this as a bug & not an enhancement.
Using your example of Project A depending on the Aspect Project J that doesn't have a Java Build state... The problem we will have is that we use the build state to record lots of information about project J. Without a correct build state it means we likely have to do FULL builds of Project A everytime. So once Project A is large enough, the performance would be unacceptable. Is there some reason why you cannot attach a Java Builder to Project J and have it build the necessary Java source folders?
The problem is that those source folders are unlikely just to contain pure java - so we can't use the Java builder, we have to use our AspectJ compiler (which is a copied and extended version of the JDT compiler). We keep the Java nature on our project because many of the other features in Eclipse do work correctly for an AspectJ project and we don't want to lose them. I'm a little rusty on where we were with this - but I think because we are a subclass of the JDT compiler we can come up with all the information that should be recorded in the state.dat file (dependencies/etc), we just don't have an interface to the state object that enables us to set it and have it persisted.
The compiler is not the only piece providing information. The Java Builder (and the AbstractImageBuilder and its subclasses) are adding info to the build state. Do you also subclass our builder classes? Why not restrict other projects that want to depend on an AspectJ project, to also be AspectJ projects?
Hi - sorry I'm slow to update this!! We have started subclassing the Java Builder code but I'm not sure when we'll get around to finishing that with all the other requirements we are getting in. It would be too restrictive to require that projects that depend on AspectJ projects also have to be AspectJ projects as we have many use cases where users gradually adopt AspectJ by gradually moving one or two of their Java projects to be AspectJ projects, rather than changing them all in one go (and indeed, as our incremental compilation support isn't currently as great as the base JDTs is, people really wouldn't want to convert them all !). Also, a built AspectJ project produces correct java class files that should be immediately consumable by other Java projects. At the moment, when AJDT is installed we switch the workbench option: Window > Preferences > Java > Compiler > Build path > "Abort build when build path errors occur" to 'off'. This is a way we can avoid Java projects failing to build because they depend on an AspectJ project (which also has the Java nature but no Java builder and hence no state.dat). Fundamentally we just want a way to avoid making that switch, so is there another way to keep a Java project happy that depends on an AJ project? Or is there some minimum amount of information that we could put into a file, call it state.dat and have the Java Project infrastructure accept it as a valid state file...
The performance will be terrible... but you could quickly subclass the JavaBuilder to always do FULL builds on AspectJ projects. Override the body of the try{...} in the method JavaBuilder.build(...) with: try { notifier.checkCancel(); initializeBuilder(); if (isWorthBuilding()) { buildAll(); ok = true; } } This would produce a state file that dependent Java projects would pick up and then also do a FULL build. If users are going to work in this mode, then they need to turn off automatic builds, so they can decide themselves when they want to take the hit. I doubt users will like this solution since it means even a change to a comment will cause a FULL build & possibly a several minute delay.
Can you explain how AspectJ projects are currectly built? Do you have an incremental builder already or are you running Ant scripts?
Imagine you have a pure java project setup in your workspace. There is a context menu available on it which says 'Convert to AspectJ'. If you select that option, we add the AJNature to the project (it will continue to have the JavaNature), we remove the JavaBuilder and replace it with our AspectJBuilder (not ant based but a proper builder subclass of IncrementalProjectBuilder) - it supports full or incremental builds. The build() method delegates to the ajc compiler, which passes its results back through a compilermonitor interface that we have and those results are translated into things like markers and problems. If at some later date you convert it back to a pure Java project, we replace our AspectJ builder with the Java builder and remove the AJNature. On the previous comment about forcing full builds, I don't think we can do that as we can have non-Java constructs in .java files in an AspectJ project and the Java builder has been removed from AspectJ projects. It would just be nice if pure java projects that depended on us didn't think we incorrectly building because we have no 'state.dat'. Is there any comparable problem with other tools that compile their own source code into bytecode? Maybe its only AspectJ that does that at the moment, but I imagine they'd have the same issue - they won't be able to create the state to keep java projects that depend on them happy.
How do your incremental builds work now? Do you use the dependencies produced by the compiler? Where are they stored? As for the proposed workaround: if you created an AspectJ builder as a subclass of the JavaBuilder & flipped the compiler to yours... it should work or at least is worth a try. "It would just be nice if pure java projects..." yes it would but we have not heard from other projects/tools with this problem. I suspect they chose to make all of their projects use the same builder. Since dependent Java projects cannot tell that an AspectJ project is not a Java Project, then the dependent Java projects expect the state file to contain all the necessary information. Half of it comes from the compiler and the other half comes from the builder. Some errors in incremental mode, like type name collisions, can only be detected by the builder. Why duplicate the builder portion?
So where are we with this?
> As for the proposed workaround: if you created an AspectJ builder as a > subclass of the JavaBuilder & flipped the compiler to yours... it should > work or at least is worth a try. When I started looking at this, I tried various ways of creating a meaningful state.dat file, one of which was making our builder a subclass of the JavaBuilder. The main problem is that the functionality surrounding the javabuilder and state is all generally default access. Creating subclasses in our AJDT plugin with the same package name as the JDT classes (in order to get round the access problems) led to classloader problems (as half expected...). At one point we were trying to make a call like this: JavaModelManager.getJavaModelManager().setLastBuiltState(currentProject, state); Which, of course, required us to subclass State - thats when the access modifiers for fields/ctor/methods in the state class caused us problems. Could the modifiers on state be changed? And then we are in charge of filling it in correctly in our subclass? I've just reread the bug and Philippe's suggestion seemed vaguely reasonable where: if a state check on a Java natured project fails but that project doesn't have a java builder (or has a disabled java builder) then you rely on that project having built correctly. (effectively putting the requirement on us to do the right thing).
Philippe's suggestion : it won't work because as soon as a prereq project had a 'valid' state we will expect to see the dependency & build info we need to perform an incremental build, but the info won't be there. So we will assume no structural changes happened to any of the changed .class files & as a result no dependent types will be recompiled. As for the workaround, we can change access to public/protected for any method you need. But I don't understand why you would need to subclass the state file. If you want dependent JavaProjects to see AspectJ projects as if they were 'normal' JavaProjects, then why change/add onto the state file at all?
We have just (last 48hours) hit the problem that will require us to create correct state files... doh! Can you possibly let me know (at a high level) what information the builders contribute to the state that the compiler doesn't? That will let me know if I should continue and finish our subclassing of the builder stuff or if I can make do with what we have done in subclassing the compiler. In our subclass of the compiler we are building a State object with a different package name to the existing one, ours is: org.aspectj.org.eclipse.jdt.internal.core.builder.State rather than org.eclipse.jdt.internal.core.builder.State I'm (thinking out loud here) imagining at some point I would need to hand back our State object to the JDT - hence if I could make ours a subclass of the JDT one that might make things easier. Thats when I'd need some visibility changes in the JDT state class. I suppose I could serialize mine and deserialize through State.read() but that seems a bit heavyweight.
Can we please get answers to a few questions that have gone unanswered: What do you need to put into a state file that is AspectJ specific? How do your incremental builds work now? Do you use the dependencies produced by the compiler in your incremental builds & where is this info stored?
I'm sorry I haven't been giving too specific answers, I've inherited this incremental compilation code and am just getting to grips with it. Let me try and spell it out: What do you need to put into a state file that is AspectJ specific? I can imagine I would want to include info about which types in a project were aspects, which classes included pointcut declarations - stuff that would mean nothing to a Java project builder but would mean something to a downstream AspectJ project that wanted to build incrementally. We also store path information in our state object, more than just classpath we have other paths: - aspectpath: specifies aspects for inclusion in the compile/weave process. - inpath: specifies pre-built classes for inclusion in the compile/weave process. How do your incremental builds work now? We have a primitive state object called AjState that lives purely in memory which records incremental state - there is only *one* at the moment so if you switch to build a second project we lose all information about incremental building your first project and have to full build (its not a great situation). After a full build the AjState records some basic information: - the list of files that were included in the build - the various paths supplied for the compile (classpath, aspectpath, etc) - which classes were built as a result of compiling which source files - some basic dependency information retrieved from ReferenceCollection When an incremental build is performed, we check if we are building the same project as before, if not then we do a full build. If we are building the same project as before, we use our incremental info to determine if an inc build is possible - by checking things like whether the paths have changed. Do you use the dependencies produced by the compiler in your incremental builds & where is this info stored? Our compiler is an extended copy of the JDT compiler with the package prefix org.aspectj added, as a by product of the compile I do believe a correct State object is built (but obviously it is an object with our org.aspectj prefix). We don't use this State object directly as we don't understand it (yet) and it doesn't have enough info in it - but we use keep some of the same data the State object seems to use (ReferenceCollection related stuff) in our AjState object. Obviously I want to be much smarter and more closely mirror the JDT - this will mean when building an AJ project I ensure the state object produced is immediately consumable by the JDT.
I have started the work to sort this out for AJDT as a number of users are starting to hit it. The best way to do it is for me to construct a state object that will keep JDT happy in determining what it needs to build. The critical information seems to be the structuralBuildTimes/structurallyChangedTypes/lastStructuralBuildTime fields as that is used when building other projects to determine how little needs to be incrementally built. Here is my initial strategy (where ajstate is our AJDT internal incremental state) - serialization isn't a great approach but I thought it would at least allow me to prototype what I need: byte[] flatState = ajState.writeOut() State jState = org.eclipse.jdt.internal.core.builder.JavaBuilder.readState(project,new DataInputStream(new ByteArrayInputStream(flatState))); JavaModelManager.getJavaModelManager().setLastBuiltState(project,jState); This works, I can serialize our state into a form that keeps the JDT basically happy - the problem is that some of the structural stuff is not stored in the serialized form: the structurallyChangedTypes structure is not included. I believe having a null value for this field indicates to dependant projects that they must full build? Now I do have the information for this field in my ajState but I can't get it into the JDT state object because the field is private and the methods tagAsStructurallyChanged() and wasStructurallyChanged(String typeName) are not visible for me to call and reconstruct the information. Any ideas what I can do? Once I am able to create a state object that will enable minimal builds of Java projects depending on an AJ project, I will look at the other end of the problem which is when an AJ project depends on a Java project. Using the state information coming into an incremental build I should be able to work out what minimal amount of an AJ project I can build.
Comments to comment 16: Your incremental story - If auto-build is on then every project is built. So your incremental story is a non-starter. Projects would constantly be rebuild from scratch on every change. The info stored from a full build is currently recorded by our State object. How much of JDT have you copied and changed the package prefix?
Comments for comment 17: Change whatever method visibility you need to... we can reconcile the changes later. It is EXTREMELY imperative that you understand how incremental builds work. Structural changes to any low level project must be propagated up to their dependent projects. I still think you should be able to throw out your build story & reuse ours (with minimal changes). If your State object extends ours & you change the compiler, what else do you need to change?
comments for comment 18: > Your incremental story - If auto-build is on then every project is built. > So your incremental story is a non-starter. Projects would constantly be > rebuild from scratch on every change. Agreed - I have just changed our system to have multiple AjState objects, one per project. > How much of JDT have you copied and changed the package prefix? We have copied jdtcore and added the prefix. We have made enough modifications to this copy to get the compiler usable from the command line and through a basic interface that we expose to IDEs. We want Eclipse to have a much better interface so we work seamlessly with existing Java projects. --- Comments for comment 19: > Change whatever method visibility you need to... we can reconcile the > changes later. I'll try and make the minimum changes I can... > It is EXTREMELY imperative that you understand how incremental builds work. > Structural changes to any low level project must be propagated up to their > dependent projects. Indeed, I can see its very important to get it right - I won't be releasing anything until I'm confident it is behaving correctly. > I still think you should be able to throw out your build story & reuse ours > (with minimal changes). If your State object extends ours & you change the > compiler, what else do you need to change? I've tried this a couple of times but had real problems due to the package naming difference, constantly transforming from an org.aspectj kind of element to an org.eclipse kind of element and back again is painful. let me see what I can do with just a couple of changes to the visiblity of bits of the JDT state class.
Just to clarify my original point: I think i was suggesting a pretty naive mode... but maybe asking too much. What I had in mind was that when the Java builder is removed, we simply treat the required project as an external classfolder (bin). We simply get deltas from the platform, when some are changed, and we react by building their dependents. But I agree that this is only a workaround, and having AspectJ jump onto our builder technology is far more appealing. But maybe my naive suggestion is a good intermediate step which could benefit to others as well (decoupling the Java builder to plug their make facility).
So how is it going? Would it help if we changed the visibility of any types/fields/methods ?
I have just (last week) released in AJDT our first stab at recognizing changes in structural dependencies when choosing whether to full or incrementally build AJ projects. Like the JDT we now remember what structural changes have occurred to classes since the last 'full' build - enabling dependees to check whether they need to do a full or incremental build. But, this is currently entirely outside of using the Eclipse State object as I'm still getting my head round the complexities of state management. My intention is to make my support behave for AJ like the plain Java support works and then look at 'setting' my data onto an Eclipse State object that Java will understand - I'm afraid I'm not quite ready to say for definite yet which bits of the State are not accessible enough for me :( However, I suspect I will need to be able to set the contents of the structuralBuildTimes structure - either by having access to the field itself via a setter or by increasing the visibility of the methods that put data into it (recordStructuralDependency). sorry I'm being so slow getting on with this...
FYI: our 3.1 release is fast approaching. If you need any visibility changes, we need to know very soon. Next month will be too late to get the changes into 3.1
Please reopen if there is anything you need us to do in future releases...
