Community
Participate
Working Groups
For the EclipseME project (http://eclipseme.org) we are looking to provide preprocessing support similar to the support provided by the Antenna Ant tasks (http://antenna.sourceforge.net/#preprocess). These preprocessor directives help J2ME developers to deal with the many differences between various constrained devices without having significant extra code where the memory is not available. The proposed Mobile Tools for Java Eclipse project can take advantage of this functionality as well. In order to maintain a development process that looks and feels equivalent to the current Eclipse workflow, it is necessary to be able to provide the compiler with source code that differs from the source code that actually exists in the file system. The only other option is to alter the actual source file with a builder that runs before the JDT compiler builder. This is a poor solution, as it will cause the source to be rewritten whenever changes to the preprocessor definitions are made, which will cause those files to be flagged as changes to source control. This bug attaches a patch to the current CVS HEAD JDT core support that provides a new "sourceCodeProviders" extension point. A source code provider can be registered to provide preprocessed or otherwise altered source code to the org.eclipse.jdt.internal.core.builder.SourceFile class when it is queried via the #getContents method. The source code providers are related to a project nature, limiting their effect to projects that can actually take advantage of the function. It would be excellent if this functionality could be considered for inclusion into the 3.2 release stream as is or with appropriate changes. I believe this is generally useful functionality built in a way that various project types can easily take advantage of it. Thanks for consideration.
Created attachment 29838 [details] Patch to provide source code providers extension to JDT Core
Kent - could compilation participants solve this one too ?
My initial look at this support in CVS makes me think the compilation participants might work. I can play with it a bit on HEAD. When is this targetted to be available in a milestone build?
I cannot see the participants helping you since its really intended for applications that generated additional source files, based on the files about to be compiled. In your case, you just want to change the source that the compiler sees. An in- place preprocessor makes more sense.
I guess I'm confused. I must have read the code wrong. It looked like I could make arbitrary changes to the AST of the compilation unit? If so, I can "comment out" the code that is not enabled via the AST before the compiler get ahold of it. Am I mis-reading this? If not, maybe the concept should be extended to allow for this to cover my case as well?
The API is not complete. The call you saw only handled the case of the reconcile operation, which is not called during the build loop. The work in progress API for the build path (not released yet) does not provide a way for you to change the source of a file. The problem with allowing you to do that is memory usage. We would need to read the source for every source file about to compiled & cache your changes in memory. For large enough projects, this would cause the VM to run out of memory. We currently only read the source for a file at a time while its parsed, not all source files involved in the build cycle.
I'm perfectly fine with not using this support for my required functionality. I'm back to believing that we need something like the source code providers I proposed in this bug. In that type of a design, my preprocessing builder can run before the compiler and output the preprocessed content to a temporary directory. Then the source code provider in this case would pull the preprocessed output for every source code request. This avoids the need to calculate the deltas more than once and keeps the source content out of memory.
Any further thoughts here? What are the next steps required to move this functionality along?
You entered the bug on Sunday. It will take more than a few days before we can find time to evaluate the patch. ;) We have quite a few things on the go already.
What? I'm not the top priority? :-) Sorry. Wasn't so much looking for a yes/no on the patch as a general "good idea" or "bad idea". Basically, in the event that something like this is not going to be accepted (my patch or or something similar), I need to figure out what that means to this functionality in EclipseME.
How does your patch affect a Java editor? Does quick fix still work? Is the Debugger happy or is it confused by line numbers being off?
That is a good question. The patch itself doesn't really dictate what the source provider does or does not do to the source code. So, it would be fairly easy to generate processed source code that has line numbers that no longer match the originals. In the case of most of the EclipseME directives, I think this can be somewhat handled by commenting out code, which should leave the line numbers alone. There is the ability to do an "include" which would most likely break line number debugging. Again, this all comes back to the source code provider. I'm not sure there is any way to avoid breakage in certain cases, but I'm open to other ideas to solve the problem. I don't think the patch as it stands makes the prospect any better or worse. I'm guessing that users of this functionality would be willing to accept certain debugging problems for the added functionality. The other side of things that I'm not really up to speed on is the source providers for the debugger. If you assume that the preprocesser runs as a builder prior to the JDT compiler and stores the processed output, it may be possible to also configure where the source is loaded by the debugger to point to the preprocessed output.
Craig, Sounds like this: The other side of things that I'm not really up to speed on is the source providers for the debugger. If you assume that the preprocesser runs as a builder prior to the JDT compiler and stores the processed output, it may be possible to also configure where the source is loaded by the debugger to point to the preprocessed output. ..Might be the way to go..the other thought is it could at a later date provide porting hooks via some AOP/aspectj process...Tira Wireless rewrote javassit to use aop to preprcoessosurce for porting beteween j2me midp implementations..Your gem of an idea is solid I think and has promise.. Fred Groitt(aka shareme)
I looked at this quite extensively over the holiday. In terms of the question about whether or not this effects the editor, the answer in my case is no. The builder run after the save is completed and the source uses Java comments to "hide" the preprocessor defines from the standard compiler. I do think there is room for additional functionality (folding, highlighting) for these syntax elements, but it is not *necessary* nor does it affect this particular enhancement request. Fred is right in terms of the debugger source providers. The way that this is currently working in my workspace is that the preprocessor builder runs before the Java compiler and writes a new copy of the file into a derived file in a subdirectory. This extension point would be used to point the Java compiler to this derived file and the debugger source providers support would be used to include this directory in a source resolution search. In this way, the user will be debugging the exact source of the file *after* processing. At this point, the only thing I have any concern with in terms of this patch is how to "safely" tie this provider to a project nature. As it stands now, multiple source code providers could potentially be attached to a single project nature. The code as it stands will choose one. In practice I don't believe this is an issue, but it is a minor hole in the extension point as currently defined. This is something that could just be documented or perhaps requires a bit more thought. Any further thoughts?
Debug commnet: as already noted, the source lookup path could be modified to retrieve the "pre-processed" source, rather than the original source. (I'm not sure if this is what developers want though). Alternately, a more complex solution would be required, where the pre-processor client would also have an opportunity to slam the debug line number table before writing the class file, in order to map back to the original source.
I think for an initial release, the current debug source support should suffice for my needs. Longer term, there may be a need/use for something more sophisticated for source lookup (if it doesn't already exist). For instance, the preprocessor I've built to match Antenna allows for an "include" directive. My current plan is that the debugger will just debug the included content within the processed file. Longer term it might be nice to actually be able to direct the debugger to lines within the original included file. In this case, the line number table won't help because the lines would actually exist in another file. Does the debug framework have anything like this? As I said, I don't think it is necessary in the short term.
Any new thoughts on this patch? Any chance it will be integrated into one of the upcoming milestones? I hate to be a pain, but I have EclipseME users that would love to have the support that depends on this. It seems if there is no more discussion, that someone should be able to say yes or no to the patch. If there is more discussion to be had or changes to be made to the support, I'd love to help make that happen.
I wonder if there could be a hook or callback that compiler will call after parsing an AST, so plugins may apply some transformations right on the AST that will be compiled.
Although I think there is room for an AST-based solution, I'm not sure that is the best solution for the current use case EclipseME has a need for. The problem with an AST-based solution is that the changes need to be recalculated and reapplied every time that the compiler needs the AST for a compilation. I haven't spent a lot of time looking at all of the paths through that code, but I would imagine the AST will be built more often than the source actually changes. In that case, having a version of the source on disk with the appropriate changes already in place should be faster than reapplying the changes to the AST over and over. Essentially, the disk becomes the cache of the processed document.
I am a user of EclipseME. I am keenly waiting for progress on this issue. I imagine the other 40 people who voted for it are also waiting for progress. I want to know why after more than 3 months, we havent had any evaluation from the Eclipse core team. The reason it is so important to us is that you cannot do commercial-level J2ME development without pre-compilation. The platforms variations are just too great, and code space is so tight that we cannot afford to ship unneeded code. With ExclipseME we have a fantastic tool that very cleverly adapts the Eclipse core to J2ME. But EclipseMe has a weight around its neck. It cannot offer Pre-processing, the most wanted & requested feature in the developer community, because of missing support in the core Eclipse platform. J2ME is an extremely fast moving industry. For those of us who have adopted Eclipse as our development environment of choice, its frustrating to sit helplessly, month after month, waiting for the powers that manage the core to consider our request. And finally, Netbeans can do it. Why cant Eclipse?
Well Ben its very simple - JDT Core has over 500 open PRs for bugs that are already part of the release. Clearly the users that want to use the patch can apply it for the short term. And as for having to wait month after month (a little exaggeration maybe?) - the patch was provided half way thru the 3.2 development cycle, when our development plan was already set.
I should add that we do think the problem is worth addressing, but we want a solution that fits in with all of the tools: the editor, code assist, debugger etc. Several teams have existing preprocessors, but they have to remember to manually edit the original source after 'fixing' it in the debugger. They accept that their solution is poor but can live with it. When we address this problem, I think we should provide them with a better solution & not just a different implementation of their existing solution.
I agree. Integration is key here. We do want a solution which can scale, where all existing tools are going to perform in a seamless fashion (think of search, codeassist as well etc...). Editor must reconcile the right source, and still deal with preprocessed code at some point. I also think as Darin that the client doesn't want to see preprocessed code in the end. This feels really close to what JSPs are doing today, and they plug into our layer, and provide some builder for it. In my original comment about compiler participant, I was hoping we could add some specific hook for preprocessing, but this did not happen by M5, due to other fights on other front. Still this is an interesting problem to tackle, but we want a consistent solution, and not an intermediate one which we will have to live with forever. Looks like we should reconsider this issue early in our 3.3 dev cycle. Craig - would you be interested in documenting exactly what is the usecase and expectations ?
To clarify, in the original description, you mostly mentionned hooking into build process, and also told about debugger. What about all the rest of the tooling ? How do you foresee developping with autobuild turned off ? In this mode, the source model is still evolving as changes are made; which means preprocessing needs to occur not just while compiling for building.
Philippe and Kent, Let me start by saying thanks for all that you do. I recognize that the Eclipse platform is a huge effort and that all requests such as this cause ripple effects. I understand that this type of functionality needs to be discussed and a plan must be put together to address all of the concerns. I would guess that some of the frustration that people, including myself, are having is the relatively low amount of discussion of this new feature. If I felt that we were talking this through and working on a plan to address the issue, I would be somewhat less concerned. I have considered providing a patched version of this functionality for my users, but I'm concerned with betting on this and having the direction change drastically. In terms of use cases, I'm not sure how much more I can say in addition to the previous discussion. I do feel that this functionality is somewhat different than something like JSP processing. In JSP processing, you start with something that looks very little like Java source files and translate that into Java source code. It is definitely more of a hybrid situation. In the preprocessing case, it seems that the user expects it to act very much like a standard Java source file. As previously discussed, there are a lot of aspects to this functionality. With that said, I think there is a spectrum ranging from "necessary" to "nice". I'm sure there are other things, but here is a list in order or "necessity" in my mind. - Ability to influence the source input to the Java compiler - Ability to influence the debugger to track this information. (I'm not sure how this would work in practice, as line number tables don't allow cross-file references as far as I know.) - Ability to influence the editor functionality. For instance, to recognize that a block of code is currently "disabled" and therefore the error analysis should not notice the variable declarations in that block. I'm sure there are many other aspects to this that I'm not thinking about at this point. I would argue that these don't have to be tackled all at one time. Adding the ability to feed the java compiler a set of source code adds value immediately, although with some potential rough edges. Those rough edges do not negate the positives that are provided by this functionality. I believe that as long as the approach for one of these features does not preclude adding the next level of functionality, it really is not necessary to solve them all at the same time. As I said previously, I'm not sure what other information I can provide in terms of use cases than can be found throughout this bug report. I'm very interested in helping to drive this forward and will provide as much information as I can. I'm just happy to have this conversation moving forward at least a little bit.
I spent some time looking at this during my flights to/from EclipseCON. It really looks to me like the compilation participant functionality may be the "correct" way to handle this. I understand there are some concerns about how to make that approach scale that are going to need to be addressed. Since it is obvious that this functionality is not going to make it into Eclipse 3.2, I'm looking for a bit of help to tide my users over until the next release of Eclipse when this is done right. In previous discussions, there was some talk of providing a "patched" version of JDT core with my functionality. I'm not a big fan of doing this due to the maintenance/upkeep of the patch. Are there any other options here? Is there some way that I can inject this functionality without having to build a patched version of JDT core? Assuming the answer to the above is no, how do I build a patched version? Is my best option to just do a plugin export or is there a better way to do this? Is there any way to conditional extend the extension point? It would be nice if EclipseME could gracefully handle the case where the user does not have the patched functionality available. I'm looking for any tips to help make this functionality work as smoothly as possible given the circumstances.
As said earlier, will queue this for 3.3
I understand that there won't be anything official until later. I'm just hoping that someone can help me do a short-term workaround.
Why did this get resolved? Why isn't it just open with a 3.3 target milestone?
Maybe allowing the patch tool as part of the build process would help. Here is the idea: Original sources are in base/. base/ offers the full Java options (classpath, navigation, etc) but the code in base/ is *not* compiled. This is under version control. patches/ contains the modification rules (standard patches). Also under version control. Lastely, there is build-src/ which contains the processed sources. This is not under version control but compiled by javac. In a second step, there could be a path which contains scripts which are applied during the build step. The important part here would be to create a list which maps the source lines of the resulting code to the source files (patches, real source, script) where they came from. In the debbugger, we'll need a view which shows the generated source and navigation help to go to the places which contributed to every line.
I guess I'm confused by Aaron's last suggestion. There have been a number of suggestions along the way (in and out of this bug). In the end, if it isn't tied directly into the JDT compiler, it isn't going to give the appropriate user experience. I've been hoping to find some time to look closer at what it would take to make the compilation participants do what needs to be done, but I've just been unable to find that time. Having looked at this, I do still believe the compilation participants support is still a better place for this functionality (changing the AST during compilation) than my original source code providers patch, but I'm willing to take advantage of either one to get to the results I'm hoping for.
Being able to hook into the AST would be even better. One of my most common problems right now is to insert code in the EMF generated Java source like here: public void setTitle(String newTitle) { String oldTitle = title; title = newTitle; if (eNotificationRequired()) eNotify(new ENotificationImpl(this, Notification.SET, WriterToolPackage.BOOK__TITLE, oldTitle, title)); } In this code, I'd like to replace eNotificationRequired() with "oldTitle == null ? title != null : !oldTitle.equals(title)". Of course, I don't want to edit the generated code because then, the code generation would cause problems (either changes in the model wouldn't be reflected or my changes would be overwritten). So the perfect solution would allow me to select the condition in the if(), raising an error if the if() isn't there anymore (so I will know when the model has changed). The problem is, as you said, the debugger. In bug 45423, we're talking about a way to map java code to a view (presentation independent of code formatting). Basically, the proposed solution is to reformat the inserted code in such a way that every part that comes from a different file ends up on a single line during compilation: if ( oldTitle == null ? title != null : !oldTitle.equals(title) ) The debugger could then use a kind of reverse map. The second solution, even more radical (and probably way beyond the scope of this bug) is to all hooks in the AST itself. The idea is to have grammar partikels in your project which explain new language constructs. So in my case instead of using EMF, I could create a new Java syntax (or extend the existing one but way beyond the possibilities of annotations): private String title { writable, notifyOnChange, notNull, maxLength=64 } This is but a simple example which could be implemented with Java 5 annotations. But how about this: HTML a = <html><head><title>Title string</title></head></html>; a.html.head.addSibling( <body /> ); String id = "xxx"; a.html.body = <div id="${id}" />; for (HTML.Div div : a.html.body.child(HTML.Div)) { System.out.println (div.id); } that is, the ability to process native HTML in Java by using the class information to determine the parser for the right hand side of an expression.
So the question for today is: can compilation participants help some preprocessors solve their problem today on the 3.2 release without any more support from JDT/Core? I think its definitely possible if the solution looks something like this: 1. Create as many 'processed' projects from a single 'original' project as you want, based on multiple tags. For example, if tags A & B are enabled then the original project X could produce processed projects X-A, X-B, X-AB. The classpath in each processed project remains the same, but each processed project is not attached to a CVS repository (unless you want to save the processed source for some reason). 2. When your compilation participant is told that the builder is about to compile a set of source files in the original project, copy the processed output of each file to its processed project (see CompilationParticipant.buildStarting(BuildContext[] files, boolean isBatch)). You can run your processor on each original source file with different tags enabled, and produce multiple results into the corresponding processed projects. Overhead of the participant should be low assuming your preprocessor is reasonably quick. 3. Then you can run/debug different targets by switching to the corresponding processed project. The source in the debugger will be the processed source and the line numbers will be correct. If you choose to make changes to the processed source, you can test fixes immediately but must remember to backport your changes to the original project. With no additional support, search will find references in code that is commented out in the original project, but visible in processed projects. Each processed project represents what you want to package for that set of tags. Source files could be as small as possible if your processor strips disabled source from other tags. Multiple independent source processors could be attached to the same original project and produce their own processed projects, without any conflicts/collisions.
I'm going to have to spend some time digesting this idea. I had begun to contemplate whether separate projects could help (thought of it in the shower this morning), so with two of us thinking in that direction, perhaps there is something there. It may be something to consider for a 3.2 base, but it is a pretty ugly hack that should likely be resolved in 3.3. I was digging around in the code that handles that to determine how hard it would be for the builders to honer the changes in an AST from the compilation participant. I also read tbe bug report in which concerns with this were expressed. What is the current thinking on that part of things for a 3.3 release? The advantage of course is that it doesn't involve new API, just new support within that API.
I don't see it as a hack at all. One alternative is to make Java projects support 'original' source folder(s) that map to several 'processed' source folder(s), each with their own class folder. But then how does the builder not blow up with 10 source files all defining the type p1.p2.X ? How does a user keep the processed results for several tags around so he can switch between targets without rebuilding ?
(In reply to comment #31) > I guess I'm confused by Aaron's last suggestion.... When we target multiple devices and also when we reuse code without unacceptably bloated OO stuctures (for limited J2ME devices) we make extensive use of the Antenna preprocessor. However we also have found the need to write our own preprocessor tasks to overcome various limitations in both Antenna and Ant itself. Although there is truly great benefit in integrating a preprocessor into the JDT the only bulletproof way of supporting all developers diverse build requirements will be to also provide hooks on which arbitrary scripts can be run at various stages of an automated build: - before the 'built-in' preprocessor - before compilation - after preprocessing - after packaging - ... Integrating Ant scripts into the automated build system of Eclipse is surely something which could be useful in many spheres of development.
After spending some time thinking about the multiple projects approach, I believe it could work but that it is not the best long term solution. In reality, if using this approach, there would be no reason for using the ICompilationParticipant support at all. I'm already able to open and parse files using the AST parser, so it would just be another builder in the chain. In the longer term, are you still considering adding the ability to alter the AST using ICompilationParticipants or something like my original request? I just don't believe the multiple project approach is appropriate as the ultimate implementation for a variety of reasons.
We have seen the multiple project solution (using separate builder) work quite well. So without specific reasons why multiple projects does not solve your problem, I don't think we will adding more support for this. Please feel free to describe why this doesn't work and include answers to the questions in comment #35.
Can you give some examples of where multiple projects are being used in this way? Are they doing things that way because they felt it was the "right" way or because that was the only way that was available to them? The multiple project approach is cumbersome and strikes me as a hack. It is significantly more difficult for plugin developers to manage multiple projects and their associated classpaths than to code to a single project. It is confusing to users, as there will be multiple projects, some of which are not being created by the users directly (this is true whether or not they can be filtered out of the user views). In short, while it seems like it could be made to work, it does not strike me as the high quality support that I've come to expect from the Eclipse platform. I don't quite understand the resistance to adding a feature like this. It is obvious from the associated vote count on this bug that I'm not the only one that cares about a feature like this. Is this a matter of resources, time or just that you don't believe it is the correct thing to do? There are a number of entries on this report that mention that they could definitely use this functionality for their plugin project as well. If you read back through this bug report, you will see that my original request was pretty simple. If the concern here is that it is too difficult or expensive to support allowing me to change things from inside an ICompilationParticipant, I'm more than willing to help look at other options including my original proposal. I just don't want to waste another year of Eclipse development without a real solution coming from JDT.
I agree with Craig that it does not make any sense from user point of view to have two projects.
I agree with Craig that it does not make any sense from user point of view to have two projects. It seems like a workaround, not a solution.
Interestingly, extra source folders for processed sources are not striking you, but extra project folders do... I agree these are more visible, but in essence they are more powerful as well. Anyhow, I think we can investigate this in 3.3 time frame. Java class library developers have similar issues when dealing with platform specific code, SWT has the same issue as well (cc'ing Steve). All took different approaches though. Things I'd like to preserve is the ability to find references. The multi project solution would show references in proper context; i.e. refs to a method in win32 as opposed to for a different platform. JDT is dealing with plain Java; we can provide hook or ease integration of preprocessors, but this isn't truly part of what Java is; and actually goes beyond just Java.
Phillipe, Thanks for your comments. I agree that this whole thing goes beyond Java. A lot of people dealing with J2ME (aka Java ME) will tell you that it is not exactly like coding in normal Java, but uses Java syntax. The set of differences and constraints in the ME world make it virtually impossible to code as "normal". On the other hand, things can and do move from the ME world over to the SE/EE world. Take, for example, the split verification support coming in Mustang. That originated in the ME world and has thus moved over the SE. I will say that I would not be quite as opposed to multiple source folders, as they are all within the context of the user's project. That said, I can think of no way to actually make that work given the curent Eclipse functionality. I may not like the multiple project approach, but I can at least envision how it might be workable.
Here the questions from comment #35 that we expected answers to: How does the builder not blow up with 10 source files all defining the type p1.p2.X ? How does a user keep the processed results for several tags around so he can switch between targets without rebuilding ? The multiple project solution is very workable and has been used by a team for several years. It allows each team member the choice of which tags should be processed... search and debugging work as expected. Given the number of votes this bug has got, I assume that someone has the time to take their processor and have it generate files in several 'generated' projects to see how it works for them, then provide feedback for everyone else.
What is the current thinking on this bug? Are there any thoughts/plans from the JDT team at this point? I don't want to see another Eclipse release slip by without progress on solving this problem. My understanding at this point of the potential solutions: 1) My original "source code providers" concept. This is straightforward and would solve the initial requirement. There may or may not be lurking "API problems" with the concept. There may or may not be a better way. 2) Expand the compilation participants functionality provided in the 3.2 timeframe to actually allow changes to pass through to the compiler. This seems like a very clean API, but I believe there are concerns with implementation and scalability of the solution. Can anyone offer more insight into concerns with the approach? 3) Multiple project approach. Although I will admit that this is likely to work, I still feel that it is not a reasonable long-term solution. Based on comments on this bug and in discussions I've had outside this bug report, I'm not alone in this thinking. I would prefer not to invest any time in this "solution" as I don't believe it is something I want to see in EclipseME or MTJ. Are there other options that should be considered, either already available or better long-term additions? I'm certainly willing to pitch in and try to help with this, but it makes no sense for me to go off and try to build something if the JDT team is going to be unwilling to accept the work when completed. I would like to see some significant design discussions before proceeding with any significant work.
Craig, you still have not answered questions that we have asked. As for the multiple project solution working vs. not working. IT DEFINITELY WORKS! We know of a team that has used this approach for several years. No one has explained why this solution is not acceptable to them or even had a co-op student try to implement it for a week.
(In reply to comment #46) > No one has explained why this solution is not acceptable to them or even had a > co-op student try to implement it for a week. One of the reasons why it i not desirable is that it effectively changes a build path for compile time and for launching. So, dependent projects has to be aware of those anomalies and launch configurations has to be ajusted accordingly. Also note that it also affet derived launch configurations, e.g. when applying AspectJ in the runtime and some other similar cases. In my opinion that is more then enough for the reason.
Ok - so how do any of the other proprosed solutions not have the same problem?
(In reply to comment #48) > Ok - so how do any of the other proprosed solutions not have the same problem? The approach I suggested supposed to preprocess source code right before compiler (e.g. after parsing to AST model). So, neither source folders or target folders would change. It may introduce some challenge for source-level debugging. Though I am not sure if those woven changes shouldbe seen during debugging.
But then you can only keep the output for 1 target for 1 group of tags. We have teams doing preprocessing that have 10-20 tags. They would like to have the output for numerous combinations of their tags kept around and built in the workspace. It is completely unacceptable to them that they have to wait for a build everytime they turn off 1 tag and turn on 2 others. So the simple approach of keeping the classpath the same and having the output folder contain the result of the preprocessed source is a waste of time for them. Their situation is no where near that simple.
Sorry - accidently tagged as closed
Tagging as later
Backing up a bit... I apologize that I had forgotten there were outstanding open questions. I should have spent the time to go back through this novel of a bug report as I just did to refresh my memory on the territory covered thus far. "But then how does the builder not blow up with 10 source files all defining the type p1.p2.X ?" I've not been considering a need for multiple tags to be "active" at one time. From a compiler perspective, there is only a single instance of the Java source at any one time. I'm not arguing that having many tags active at one time would be a bad thing, but being able to have a single tag active is better than what developers have right now. I think doing "batch builds" of multiple "configuration" is better kept to Ant and automated builds. In the end, the ability to provide a "virtual file system" view of the sources does not imply to me a particular use for that functionality. "How does a user keep the processed results for several tags around so he can switch between targets without rebuilding ?" Again, this strikes me as something that should be up to the user of the API to determine. If the plugin chooses to manage the results as separate project, separate derived folder or somewhere completely outside the workspace it should be of no concern to the builder/compiler. We seem to be hung up on the issue of using multiple projects versus some other solution. I understand how that can be done and believe that it is technically doable. I *personally* believe that this is wrong from a user's perspective. I *personally* believe that projects are first class user interface objects that are to be managed by the user, not created implicitly. I'm not a particularly big fan of tools such as TPTP creating projects on the fly to store their data either. This is a purely *personal* opinion, but one that it sounds like others agree with. As previously mentioned, I do not have a problem with creating project-level derived resources, as they can be managed and to some extent hidden from the user. I do not believe that there is a significant difference in the amount of work necessary for any solution to be implemented on my part. EclipseME will need to manage the processed source in multiple projects or somewhere else. It is an issue that EclipseME has to deal with. Given a generic interface into the compiler source, how EclipseME chooses to manage processed source should be of no concern to the compiler. This strikes me as a good thing. Please note that I mentioned "virtual file system" earlier. After some discussions with others, I'm beginning to consider a possibility of using EFS within a single project to manage the source provided to the compiler. If this works, it would be the generic hook that I need and no changes would be necessary to JDT. The trick is in signaling to my EFS implementation that this is the builder running and not the editor (to give out the right sources). One potential here is to set a thread local variable around the build functionality. For this to work, JDT would have to be single-threaded. Can anyone verify that the JavaBuilder, image builders and low-level compiler run within a single thread or are they multithreaded? What other issues might exist with this approach?
There is more than just the compiler to think about. JDT also needs to consider how search behaves and does the debugger see the correct source for the .class files, etc. Ignoring the problem of multiple tags generating different source likely means the solution is a waste of time for a large percentage of the listeners to this bug. Assuming your source looks anything like: /*<MY-PREPROCESSOR> IF A */ x.doSomething(); /*<MY-PREPROCESSOR> ELSEIF B x.doSomethingElse(); /*<MY-PREPROCESSOR> ELSE C x.doNothing(); */ How do you expect search to find senders of the method doSomethingElse() if the only generated source was with the tag A enabled and never the tag B or C? As for your question: at this time the builder is single threaded, but there is no quarantee that will continue. The indexing part of search is run in its own background thread while queries are permitted from any thread.
IMHO, the first step here would be a way to get the AST before the compiler/index thread sees it. This would allow people to write arbitrary preprocessors (= AST-to-AST transformers). The next step would be to see which ideas would evolve from that. AspectJ didn't happen in a day. The idea was there, then the compiler but it took several years for people to find out what works and what doesn't. Right now, we all agree that Java is somewhat limited but we cannot agree how to solve that. So lets add an extension point which allows to add A2A transformers and let's compete for the best idea how to use this EP to make us all more productive. I bet that this EP will trigger a plethora of plugins which will do things with Java that we can't even start to imagine right now.
(In reply to comment #42) > Anyhow, I think we can investigate this in 3.3 time frame. > Java class library developers have similar issues when dealing with platform Hi Philippe :-) Any chances that you could look into that for 3.3 per your prior comment? This is one of the most popular bugs....53 votes ... It is definitely is the bugzilla TOP 10! Any level of support there even if it is not perfect would be really cool. I am re-opening that bug too. Cordially
Well I cannot re-open it... Craig or Kent? Could you do that please?
Could we at the minimu provide with some decent extensibility via fragments? like o.e.ui.workbench does? It would be as simple as adding the following directive to the manifest : Bundle-ClassPath: . This would allow frgamnets to provide patched code when needed... and would go a long way to support many interesting experimentations.
Sorry my mistake: the directive should be something like: It would be as simple as adding the following directive to the manifest : Bundle-ClassPath: extensions.jar, . where extensions.jar could be optionally provided by a fragment is would not be in jdt.core . Any code in extensions.jar would be first in the jdt.core classpath :-)
There have been a lot of discussions throughout the lifetime of this bug concerning possible workarounds to solve this problem. The primary suggestion has been to use two separate projects to solve this problem. In fact, EclipseME is currently shipping a version of its preprocessor functionality using this approach. Unfortunately, this approach has proven to be as much of a problem from a user's perspective as predicted. This is in addition to the development effort that was required to get this approach to even work at all. There are currently a number of EclipseME bugs that have been written that likely can never be solved using this approach. Two simple examples: * Java search returns results from both the primary and secondary projects. * As a Java project, the secondary project shows up in the clean dialog. At this time, I'm exploring the possibility of using the OSGi Framework class loader hooks to rewrite the JDT SourceFile class bytes during load. It is certainly dangerous to rewrite classes in the face of potential change, but it appears to be absolutely necessary to maintain a reasonable user workflow without further help from the JDT team. Although I would much prefer to use a sanctioned hook into the compiler, I will go ahead with my plan to hook the framework classloader. As always, I am more than willing to participate in design and implementation of a sanctioned extension point. EclipseME users would likely benefit from this in the long term. As part of the Mobile Tools for the Java Platform team, I can also say that MTJ would prefer to build on a sanctioned extension mechanism. I am well aware that such a hook does not solve all of the problems raised by this bug. Multiple device support, debugger source handling and editor support are all examples of functionality that will also need to be addressed at some point in the near future. Most, if not all, can be solved with other extension points available in the platform. But, without a hook into the compiler none of these matter. Please take one more look at this bug and consider what might be able to be addressed in the 3.3 timeframe for the benefit of the users of EclipseME, MTJ and everyone that has voted for this bug. If there has been progress or discussion about a solution for this bug, please update the bug to reflect that progress. Thanks.
I'm removing myself from this bug report. I can't be of any help and have long since resigned myself to accepting the limitations of Java and its effects on the tooling. See ya!
I just had a really stupid idea but ... well, see for yourself: Is it possible to call the Java compiler from a classloader? Basically, I would feed the classloader a classname. Certain properties (for example, the package) would hint it to generate the code, then run it through javac and finally return the generated class. I agree, this would be slow but that's not the point. We could add caching and for the final product, we could resolve all these classes once and pack the generated code in the JAR. My main point is that this would give us a well defined (as far as classloaders go ;-) point to hook into Java execution. Comments?
Aaron, re: class loader: Frankly, the idea sounds unworkable and rather off-target. We're trying to deliver pre-processed source to a compiler here. The class loading happens on a phone, much later.
Aaron, I think I understand your suggested approach and that you are suggesting to do the classloaders from within the Eclipse side of things. While it might be possible to do, it is definitely not the path that I want to see things go. I'm working hard to make sure that the user's experience is as close to the standard JDT experience as possible. Calling out to javac will not yield the tight integration that I'm looking for. In reality, calling out to javac could easily be done without any classloader tricks from within a standard Java builder.
Here is an idea borrowed from NetBeans mobility pack preprocessing support: instead of hooking into the build process, the code itself is modified, but in a "toggle-able" way. specifically, in NetBeans they have a Commenting-Preprocessor, that comment and uncomment blocks of code, based on the current profile (set of define variables). in NetBeans, the user can initiate a re-commenting of the code by pressing a key combination. the code is also re-commented after the user changes the defines. a concrete example: //#if FOO System.out.println("Foo is defined"); //#elseif System.out.println("Foo is not defined"); //#endif when the re-comment command would run, it will prefix the first line with //# comment. if FOO is then defined, re-comment will run again (automatically) and the first block would be uncommented and the second would be commented. the main problem with this approach is the interaction with the version control: since the code is actually changing, it will be marked as changed for the purpose of version control, which is problematic - but personally I would live with that. On a side note: I have seen the multiple-project approach offered by EclipseME and I agree with Craig that is a bad solution from the user's perspective.
I'm trying very hard to avoid things that mess with source control. It is bad enough with something like CVS and SVN that do no locking, but for systems that use locking it would be a nightmare in my mind.
We should investigate this enhancement for 3.5. Kent, for 3.5M3 please define the full requirement across the board: - how does it interact with the compiler/builder? - how does the debugger map the source to what's being executed? (need mapping api?) - how does it get presented in UI (multi-tab editor for toggling filters?) ? - what impact does it have on JDT tooling (model APIs, search, etc.) ? - what extension points are needed? - more ?
So to everyone that voted for this & to those that have already commented - please take the time in the next 2 weeks to let us know what you would like to see in 3.5, since it has been a long time since this bug was originally opened. If you could send an email directly to me ( kent_johnson@ca.ibm.com ) & I'll summarize the comments when I'm back in 2 weeks. If you have written a preprocessor, please let us know : - did you attach it to eclipse ? how ? - approx. how many source files need to be preprocessed ? - is performance an issue for you or is preprocessing 'real fast' ? - did you 'cache' the processed source ? - how many tags does your source contain ? - do you generate/debug several different targets at the same time ? - how do you expect the debugger/search/code assist to deal with the original and processed source ? - and anything else you can think of...
- did you attach it to eclipse ? how ? Yes. my preprocessor (the one which will be used with MTJ) is reversible, meaning you can execute it again on a previously preprocessed file and get correct results even if you changed the symbols between the two runs. I used this property, what I did was simply to edit the text in the editor buffer. since my preprocessor prefix 'hidden' code lines by a Java comment the code in the text editor compiles and does not confuse the syntax checker. - approx. how many source files need to be preprocessed ? depends on the project. when a file is modified (saved) it's preprocessed. when the entire project is rebuilt, all the files are preprocessed. (I integrated as a builder). - is performance an issue for you or is preprocessing 'real fast' ? I am not sure how much of an issue it is. it wasn't an issue for me. - did you 'cache' the processed source ? Yes, in the editor buffer (meaning on top of the actual files, this is not what you would normally call cache, but it has the same effect: a file is preprocessed only when it needed to be). - how many tags does your source contain ? The terminology is symbols. it can be anything really, my plugin uses a device database which can populate the symbols, and the user can add his own one per project, or per device through an external xml file that extends the device database. - do you generate/debug several different targets at the same time ? No. I consider that the job of the build system (IE: Ant + Antenna, or something similar). - how do you expect the debugger/search/code assist to deal with the original and processed source ? At any given time, there is a set of active symbols per project that have preprocessing enabled. debugging, search and code assist should work on the code which have been preprocessed with those symbols. - and anything else you can think of... Syntax highlighting show show what code is active and what code is 'hidden'. since the entire point of this bug report was to eliminate the need for my approach by being able to provide the preprocessed code to the compiler, I think the original source should not be modified (as it was in my plugin), but instead preprocessed to a different dir (much like classes are compile to bin). assuming this is how things will be, it will be hard for the user to understand what code is active and what code is not. the syntax highlighter should take the preprocessor output into consideration and paint hidden code in disabled colors to make it clear to the user that it's not active. to achieve this, the best way would be to keep an additional data model per file that hold output from the preprocessor about which lines (numbers) are active and which lines (numbers) are disabled after the last preprocessing run. this may require some changes to the Preprocessor.
I want to make it very clear that while the underlying preprocessing engine that translates the preprocessing directives into he output source control is indeed the engine that Omry has written and is embedded in Antenna, Omry's plugin, EclipseME and MTJ. *HOWEVER* the way that it is attached and used within Omry's plugin and EclipseME/MTJ is very different. Both are options to be looked at, but it is important to keep both of them separated. My comments are related to EclipseME/MJT, as MTJ is now a derivative of EclipseME. - did you attach it to eclipse ? how ? Yes. Via a an OSGi bundle hook on the bundle classloader. The JDT org.eclipse.jdt.internal.core.builder.SourceFile getContents() method is rewritten to consult a SourceMapper interface. This interface allows a new resource to be returned to the compiler. The code for this hook can be found in MTJ source at https://dev.eclipse.org/svnroot/dsdp/org.eclipse.mtj/trunk/plugins/org.eclipse.mtj.core.hooks/src . The key here is that the code is processed into a different directory than the source directory to avoid altering the user's source, as this avoids issues with source control showing changed files that have not really changed. - approx. how many source files need to be preprocessed ? As Omry says, it depends on the application. Mobile applications tend to be much smaller than SE/EE applications. - is performance an issue for you or is preprocessing 'real fast' ? - did you 'cache' the processed source ? Performance does not strike me as the primary issue. At least for EclipseME/MTJ the preprocessed results are only generated when a file is altered and saved. - how many tags does your source contain ? As Omry says, this will vary depending on the source used. - do you generate/debug several different targets at the same time ? At this time, no. Although that would be something that might be interesting in the longer term. - how do you expect the debugger/search/code assist to deal with the original and processed source ? I would expect the functionality to primary work based on the processed code. - and anything else you can think of... As I've stated multiple times in the past, there are lots of ways this could be approached and multiple levels of integration. While I'd love to see all parts of the JDT functionality addressed in one fell swoop, I would prefer to see a plan that breaks this functionality and addresses it piece by piece. The first and foremost need is a hook similar to what is currently being done in EclipseME/MTJ by hooking the JDT compiler and rewriting its code. It DOES NOT solve all of the problems, but it solves the MOST IMPORTANT highest priority problem. Waiting until JDT can address every single feature and function makes me nervous whether anything will get done.
To be honest, I was hoping for more feedback from the 57 voters of this bug. Other than EclipseME/MJT, are there other existing/planned preprocessors that would like to add their 2 cents to the discussion of what we try to accomplish for 3.5 ? Please let us know as soon as you can. thx
My comments above (#30, #32 and #55) still apply: I'd like a way to invoke the patch tool and to do AST transformations during the build.
I just wanted to clarify answers to this question if I can. - how do you expect the debugger/search/code assist to deal with the original and processed source ? Omry you said : > At any given time, there is a set of active symbols per project that have > preprocessing enabled. > debugging, search and code assist should work on the code which have been > preprocessed with those symbols. Craig : > I would expect the functionality to primary work based on the processed code. The debugger obviously needs to show the processed source so the line numbers match up. But since the user likely does NOT want to edit the processed source, are you sure that you want search to find references relative to the processed source and not the original source ? Assuming the processed source is never committed to CVS/SVN, shouldn't the user edit/search the original source ?
Interesting and somewhat difficult question to answer... In terms of editing, I think you are always correct... you always want to edit the original source code. The preprocessed source should not be editable. I think search is more difficult to answer. Take a "find references" search. If you search across the original source file without taking into account the current "processing symbols" the user may find references that are not *currently* active. It also leads to an interesting question about refactorings and such.
I agree it is interesting... it makes me think there are 2 very different solutions for this problem in the short term. **** Let's first assume that we're NOT trying to provide a HUGE solution, but a fairly simple one that people can digest easily **** I see 2 choices: 1. So IF we decide that the original source is the source we want displayed in the editor & used by search/code assist. AND the processed source is fed to the compiler/debugger. THEN a preprocessor extension point similar to a CompilerParticipant is the logical solution. Only the compiler/builder/debugger would be made aware of this 'hook'. 2. But IF we decide that the original source is the source we want displayed in the editor. AND the processed source is fed to the compiler/debugger and also the search indexer, and code assist [Not sure that refactoring will work in this mode]. THEN a new kind of project is the best solution. It would add onto a JavaProject with the notion of an 'original' source folder, to go along with a 'normal' source folder. Each file (do not care whether its .java or .jpg) would be passed to a user supplied Processor and saved in the equivalent place in the JavaProject's 'normal' source folder. This transformation/processing step would happen whenever an individual file was changed, but NOT during a build loop. Instead a normal resource listener would be responsible for noticing when the file changed. There would be no reason to change the compiler/builder/debugger/search/etc since it would appear that the processed source in the 'normal' source folder is the active source. We would need to make sure that the original source is committed to the repository and the files in the JDT source folder is not. Make sense ?
hi kent, looking at the options, it seems to me that option 1 is the easiest way to go and also matches the original requirements of this bug. but option 2 seems to be the right way of doing that and also avoid the issue raised by craig of feeding the the code assist with possibly "invalid code". so... if there is is time to do it and you think it is a good solution, i would prefer option 2. :) gustavo (In reply to comment #75) > I agree it is interesting... it makes me think there are 2 very different > solutions for this problem in the short term. > > **** Let's first assume that we're NOT trying to provide a HUGE solution, but a > fairly simple one that people can digest easily **** > > I see 2 choices: > > 1. So IF we decide that the original source is the source we want displayed in > the editor & used by search/code assist. AND the processed source is fed to the > compiler/debugger. > > THEN a preprocessor extension point similar to a CompilerParticipant is the > logical solution. Only the compiler/builder/debugger would be made aware of > this 'hook'. > > > 2. But IF we decide that the original source is the source we want displayed in > the editor. AND the processed source is fed to the compiler/debugger and also > the search indexer, and code assist [Not sure that refactoring will work in > this mode]. > > THEN a new kind of project is the best solution. It would add onto a > JavaProject with the notion of an 'original' source folder, to go along with a > 'normal' source folder. Each file (do not care whether its .java or .jpg) would > be passed to a user supplied Processor and saved in the equivalent place in the > JavaProject's 'normal' source folder. This transformation/processing step would > happen whenever an individual file was changed, but NOT during a build loop. > Instead a normal resource listener would be responsible for noticing when the > file changed. > > There would be no reason to change the compiler/builder/debugger/search/etc > since it would appear that the processed source in the 'normal' source folder > is the active source. > > We would need to make sure that the original source is committed to the > repository and the files in the JDT source folder is not. > > > Make sense ? >
I don't have much time but I had an idea how to solve the "real/processed source debugging" problem. We need these features: 1. A "#line" statement which the compiler translates into a "debug source location" bytecode. 2. A source preprocessor must be able to ask the compiler to put each node of the AST in a different line and generate a mapping tables which translates a line/column information in the real source into a byte code position in the compiled code. 3. A stacktrace translator which maps byte code positions using the translation tables into source positions where each position may be in one of three places: The real source (which the user wrote), generated code (which can be around stuff in the real source or injected between stuff the user wrote) and support byte code (generated byte code, for example with the asm framework; calling library code is not an issue because that code has correct line number information). 4. A flag in the compiled bytecode which the Eclipse debugger looks for which enables the translation. 5. Maybe a source viewer which can switch between the original source, the intermediate source (as generated by the preprocessor) and a bytecode decompiler.
Kent, Can you explain how your #2 solution would work? Two source folders or are you talking about a single source folder? If it is a single source folder that is being managed by the user *and* the builder, I think this would be hard to manage since the user may want to make changes that would need to be allowed back to source control, while other changes would not go to source control. If you are talking about two source folders, how would you picture those being managed?
It would be 2 separate folders & the second one with the processed files would be tagged as 'derived' & read-only (as would all of its files). This would make these 'generated' folders/files not visible to source code control, just as .class files do not appear in outgoing changes. So we're essentially tricking a JavaProject to think that this generated source folder is the real one & treat it as a normal source folder (ie. compile its processed source files to .class files in the bin folder, index & search its processed source files & display their contents in the debugger). The 'original' folder would be treated as a normal source folder is now. All of its resource changes would appear to be outgoing changes to CVS, etc. So a simple 'PreProcessor enabled JavaProject' would look like: Project1 bin [derived folder not visible to source control] processedSrc [derived & read-only folder not visible to source control] originalSrc [all files would be 'processed' & result saved to processedSrc] The tricky parts to be worked out would be: - how to hide the processedSrc folder from the UI in the packageExplorer - opening the 'correct' source file for certain situations such as from a search result (want the read-only processed file), or the open type dialog (likely want the original file). - attaching the correct icon to the 'original' source folder in the package explorer instead of a random folder (if we care about UI issues like this) So one question I have is : how useful is this solution to multiple users/applications ? Or should MTJ provide their own customized JavaProject that provides a hook for their preprocessor & manage their symbols ?
> - how to hide the processedSrc folder from the UI in the > packageExplorer I'm not sure this is a good idea. There should be a simple way to see the processed sources. I was wondering if it was possible to add a "source/processed" tab (like in an XML or HTML editor) to the Java editor. In the processed tab, you could see immediately what happens to the code you just typed. It would be great if that wasn't limited to Java editors but a standard feature of code editors for processors which aren't limited to Java.
Kent, I do believe there is relevance to a solution outside of MTJ. Perhaps there is a way to find out from other users? In terms of a technical approach, the approach you describe seems like it might work. I do believe it needs to be possible to choose to view the processed sources in some cases. It should also be possible to filter the folder(s). However, I don't believe it is necessary to have multiple tabs in the UI to support this... Finally, There is now a discussion going on within the MTJ project to add Multi-configuration support to the tooling. The discussion can be found at http://wiki.eclipse.org/DSDP/MTJ/Requirements/Multi-Configuration_Support . Given that this support only provides for a single "active" configuration, I don't know that it affects this discussion too much. It does imply that changing the active configuration would need to rebuild the processed source and associated binaries. Does that also imply rebuilding the search indices or anything else? (NOTE: It was already possible to switch the active configuration, but it will now be more obvious)
one thing that is not clear to me is if this is useful outside the scope of java code. if it is then the solution to extend a javaproject might not be the correct one. (but this bug is open on JDT, so maybe we can just focus on java). but maybe, as craig said, we should see if others on the list have comments. in terms of mtj, this solution is ok for us. other question that i have is about the synmbols management. currently we do the symbols management inside mtj. on this solution you expect that this management will be moved to JDT? so jdt would provide an api to create symbol sets and add symbols to it? thanks gustavo (In reply to comment #81) > Kent, > > I do believe there is relevance to a solution outside of MTJ. Perhaps there is > a way to find out from other users? > > In terms of a technical approach, the approach you describe seems like it might > work. I do believe it needs to be possible to choose to view the processed > sources in some cases. It should also be possible to filter the folder(s). > However, I don't believe it is necessary to have multiple tabs in the UI to > support this... > > Finally, There is now a discussion going on within the MTJ project to add > Multi-configuration support to the tooling. The discussion can be found at > http://wiki.eclipse.org/DSDP/MTJ/Requirements/Multi-Configuration_Support . > Given that this support only provides for a single "active" configuration, I > don't know that it affects this discussion too much. It does imply that > changing the active configuration would need to rebuild the processed source > and associated binaries. Does that also imply rebuilding the search indices or > anything else? (NOTE: It was already possible to switch the active > configuration, but it will now be more obvious) >
Making the processed source available (preferably as read-only) is definitely reasonable - but likely users would have to be instructed how to find it. As for the symbol management - no I wasn't think it would become part of our support. Right now I think the Preprocessor & its inner workings should be left up to each individual project. Possibly down the road we can generalize a solution, but I'm not sure we could get agreement right now. So I was thinking more about search. Assume an original file looked this: class X { void foo() { // #ifdef My_PROJECT call_function(1); // else // call_quick_function(2); } } And the processed source looked like this : class X { void foo() { // #ifdef My_PROJECT //call_function(1); // else call_quick_function(2); } } Search would record a reference to call_quick_function from X and would select the call in the read-only processed file correctly... BUT what if we want the search result to highlight the call in the original file? Will the line/character positions be correct ? My guess is no, they would be off a few lines/characters especially if the preprocessor removes commented lines. So if opening the processed file is fine, then we're OK. If not, we would have some work to do.
for me, what you suggested is o. one question. how i would set the list of "active" symbols in one specific preprocessed project? (In reply to comment #83) > Making the processed source available (preferably as read-only) is definitely > reasonable - but likely users would have to be instructed how to find it. > > As for the symbol management - no I wasn't think it would become part of our > support. Right now I think the Preprocessor & its inner workings should be left > up to each individual project. > > Possibly down the road we can generalize a solution, but I'm not sure we could > get agreement right now. > > > So I was thinking more about search. Assume an original file looked this: > > class X { > void foo() { > // #ifdef My_PROJECT > call_function(1); > // else > // call_quick_function(2); > } > } > > And the processed source looked like this : > > class X { > void foo() { > // #ifdef My_PROJECT > //call_function(1); > // else > call_quick_function(2); > } > } > > Search would record a reference to call_quick_function from X and would select > the call in the read-only processed file correctly... BUT what if we want the > search result to highlight the call in the original file? > > Will the line/character positions be correct ? My guess is no, they would be > off a few lines/characters especially if the preprocessor removes commented > lines. > > So if opening the processed file is fine, then we're OK. If not, we would have > some work to do. >
> how i would set the list of "active" symbols in one specific preprocessed > project? Whoever supplies the preprocessor needs to provide a way for users to add symbols & choose the active one(s). We do not have enough feedback from several implementors to know what a good general solution would be.
I agree that it makes the most sense for MTJ to manage the symbols and activation of those symbols. I also believe that it makes the most sense for MTJ to provide the implementation of the "preprocessing language". I think where it might be useful to provide platform support is in making this all possible in a generic way... be it through an extension point or a specialization of the Java project structure. That way anyone that wanted to could provide preprocessing types of activities without having to build out the low-level platform hooks.
hi craig, i thought that the preprocessing language itself (syntax of the preprocessing) would already be implemented inside JDT. we just need to provide the set of active symbols. am i missing something? my understanding was that we would only need an api to set the symbols and maybe also the types of each symbols (necessary to validate the expressions). for example, we could have a set of four active symbols: <symbol, type, value> screenwidth, int, 270 screenheight, int, 256 devicemovel, string, nokiaserie40 JSR179, boolean, true //if (screenwidth > 200) Image.createImage(big_image) //else Image.createImage(small_image) //endif based on that, the preprocessor would generate the final preprocessed code and also implement some editor features such as code competition, validate expressions, etc. (In reply to comment #86) > I agree that it makes the most sense for MTJ to manage the symbols and > activation of those symbols. I also believe that it makes the most sense for > MTJ to provide the implementation of the "preprocessing language". I think > where it might be useful to provide platform support is in making this all > possible in a generic way... be it through an extension point or a > specialization of the Java project structure. That way anyone that wanted to > could provide preprocessing types of activities without having to build out the > low-level platform hooks. >
Gustavo, JDT will only provide you with the extension point to attach your preprocessor. We do not know anything about your symbols or the syntax that you chose to write your preprocessed code. That is all left up to individual Preprocessors to handle themselves.
I agree with this approach. There are potentially multiple types of preprocessor languages that could be applied... JDT should not be required to know all of those.
so let me see if i understand: - currently we have a classpath hook that requires the config.ini to be changed - this is not going to be necessary since we will use another extension point provided by JDT. so we will keep our current solution on mtj, but use an extension point instead of the classpath hook - JDT will not define a new project type. just this extension point that can be used by anyone that wants to use preprocessing - the client of the extension point must define all preprocessing related tasks (language) is that right? other question. is the editor going to be changed to do that fold in/out, code assist, evaluate expressions, etc? i just want to understand clearly what will be provided and what we have to provide on mtj :) gustavo (In reply to comment #89) > I agree with this approach. There are potentially multiple types of > preprocessor languages that could be applied... JDT should not be required to > know all of those. >
Gustavo, what we're suggesting is : - an extension point for any Preprocessor - an expanded JavaProject (adds an 'OriginalSrc' folder) - a resourceListener that notices whenever a file changes in the OriginalSrc folder & calls the attached Preprocessor, then saves the processed file to the normal source folder & marks it as derived/read-only so CVS will ignore it - debug & search will use the processed files in the normal source folder What is NOT provided is : - what the original source looks like; individual preprocessors can use whatever syntax they want - support for symbols and how the current symbol set is identified As for changes to the editor - not sure exactly what you had in mind. A user can always open the processed source file to see what source was compiled.
hi kent, thanks for the answers. it is clear to me now. about the editor, i was wondering about a code assist on the possible symbols that are available, but since the preprocessor will be defined outside jdt, then it doesn't make sense to have that inside it. base on what you answered, probably it will not be necessary to set the current active symbols in JDT isn't it? since the component that implement the extension will provided the final preprocessed code, JDT don't need to be aware of the symbols that are active. is that right? thanks gustavo (In reply to comment #91) > Gustavo, what we're suggesting is : > > - an extension point for any Preprocessor > - an expanded JavaProject (adds an 'OriginalSrc' folder) > - a resourceListener that notices whenever a file changes in the OriginalSrc > folder & calls the attached Preprocessor, then saves the processed file to the > normal source folder & marks it as derived/read-only so CVS will ignore it > - debug & search will use the processed files in the normal source folder > > > What is NOT provided is : > > - what the original source looks like; individual preprocessors can use > whatever syntax they want > - support for symbols and how the current symbol set is identified > > > As for changes to the editor - not sure exactly what you had in mind. A user > can always open the processed source file to see what source was compiled. >
Kent, what's the status on this bug? Any hope to have something for M5?
Yes...
We've just had a user report that the bundle hook that we have been using for doing this work in past releases no longer works for the 3.5 stream. If there is any chance that the JDT work is not going to be there in the 3.5 timeframe, the MTJ project needs to know immediately so that we can look at what other options may be available.
Can I add my 2 cents to the already long discussion: * The preprocessed source code is fed to the
Sorry for the half-baked comment... Continue with your discussion:)
Moving to M7.
hi oliver, my name is gustavo and i'm one of DSDP MTJ commiters. we plan to use thie preprocessor feature on MTJ 1.0 that will also be released with galileo train. do you have some nightly build that we can start working on top of it? or at least some design that we can take a look to have an idea of what we need to do on our side to useit. i'm concerned that we don't have enought time to do our part if we are only able to start to work after M7. if there is anything that we can do to help you guys, please let us know thanks gustavo (In reply to comment #98) > Moving to M7. >
Created attachment 128768 [details] Proposed preprocessor example Gustavo, Craig and anyone else still listening: This patch contains an example project with a builder that calls a registered preprocessor (attached thru an extension point). The preprocessor controls when full vs. incremental builds occur & is asked to process each resource in an 'original' folder. The processed contents are written to a 'processed' folder, which in the case of JavaProjects would be on its classpath. The patch is not specific to JDT, but a JDT extension is provided to help in testing. If you're not interested in the simple tests, then you can remove the project dependencies to the 4 test projects: org.junit and org.eclipse.jdt.core.tests.* Now for the 'not-so-good-news' : the powers that be have decided that this patch will not released as part of 3.5. It is being provided here so users can have a good starting point on their own preprocessor infrastructure. It should be sufficient so teams will only need to define their own preprocessor & not all the 'extras' to plug it into eclipse. Please let me know if there is anything in the example project which is unclear.
We have waiting for this so long. Even if this patch does not come as part of 3.5, will it come as part of an upcoming release?
At least in terms of MTJ, we are still trying to determine if this patch really solves the problem that we set out to solve. What is your use case?
We won't do more for 3.5
