Community
Participate
Working Groups
N20050622-0010 AFAIK, the plugin exporter currently builds a plugin's class files from scratch before it packages the whole plugin. What I request is an option to tell the pde exporter to build the plugin with the normal eclipse build mechanisms, and then take the contents of the output ("bin") folders to package the plugin. This would speed up plugin-exports quite a bit, and it would probably solve bug 66177 and bug 101229.
are you actually encountering a problem here (e.g., bug 66177) or is this a performance issue?
For me it's performance. We currently use our own scripts to do our daily plug-in export for self-hosting on all our ZRH source plug-ins. These scripts use the output folder and hence are much faster than calling the ant scripts generated by PDE Export. Due to the new '.qualifier' in the version we'd like to move away from our custom export scripts and not re-implement the version patching. I guess for Markus it's also to have a resolution for bug 66177.
I personally don't suffer from bug 66177. More important for me are bug 101229 (which is marked as 'unfixable') and also performance as in Dani's comment 2.
To me, the performance argument is not the most compelling one. I can of course see how much faster it would be since you do not have to recompile but still. As for bug #101229, I think it will eventually be fixed since new support has been added to pde build to compile each plug-in against a specific JRE.
I believe this could be done by PDE/UI with no changes in PDE/Build. They would just skip the call to run the build.xml script using the build.jars target (they still may need to call the build.sources target). They would then need to collect the existing .class files into the library jars or the @dot folder. Then they proceed with running the assembly scripts as normal.
re comment 5: Andrew, I'm not sure about pde/build internals, but I have always gotten the impression from Pascal that it's much harder than what you describe there. Otherwise, it would have done it a long time ago, particularly to address bug 69620
As an additional comment the issue for me isn't performance but consistency / principle of least surprise. That is, it creates a situation very often that runtime testing and building for plugin operation isn't consistent with exporting. I am not oppsoed at all to ant being the principle build mechanism -- I've always used ant in the past -- it just seems odd to me that we have all of the support for pde builds in Eclipse proper and yet there is an additional *required* step. If there are sound reasons to requiring ant, and I can well believe there are, then there should be a functional requirement (pulling out my old SE textbook here ;)) that the ant build behaves exactly the same as the default build with respect to all PDE configuration. With #66177 that is clearly not the case, so either 66177 should be "fixed" _or_ linked folders should be disallowed from any PDE builds.
I think this should be reinvestigated and considered for 3.4
> I think this should be reinvestigated and considered for 3.4 Yes, please. Every time I try to export jdt.ui using PDE, I have to cancel it after a few minutes, because I see no progress, and the heap monitor indicates that I'll soon run out of memory. Our scripts do the job in 10 seconds. This feature is unusable for me (except for toy plug-ins).
I completely agree with my good friend Markus (comment 9).
I too agree with comment 9 except for the implication that all plugins not as massive as JDT UI are toys. less is the new more Markus ;-) Also, it is interesting that at the recent Equinox Summit Patrick Dempsey (newly CC'd here) had some compelling usecases that may fit nicely into the p2 story...
It would be nice to get this for 3.5 (the sooner the better ;-).
P2 made this basically a blocker for us. Before, we could do a plug-in export with simple ant scripts in under 2 minutes, put the plug-ins into plugins/ and Eclipse took the latest versions. Now, we need a feature patch (bug 222945), and have to wait ages until PDE has compiled everything again.
Could you please set a target milestone and make sure this makes it into the PDE/Build plan? With p2, our simple export scripts can't work any more, and the apparent lack of daily self-hosting in most teams indicate that doing daily plug-in exports is currently just too cumbersome for most developers.
I agree that the fact that the current PDE export is slow is not great, but I disagree that it is a blocker. I can load the JDT plug-ins and export them successfully.
>I agree that the fact that the current PDE export is slow is not great, but I >disagree that it is a blocker. I can load the JDT plug-ins and export them >successfully. Good for you. Maybe we should add another severity called 'unusable'.
I would like to see this improved asap as well. Here we simply no longer patch, as too painful. We instead run runtime workbench which is less practical... so basically we lost a nice characteristic, and worked around it.
(In reply to comment #16) > Good for you. Maybe we should add another severity called 'unusable'. > Now, Dani. Play nice. ;-) Is this being considered for 3.5? (Yes, I can see the target milestone)
This will depend on the project being export having correct entries in its build.properties file. I have raised bug 253950 The required property is the output.<library> property (commonly output..). This property is expected to specify the output folders corresponding to the source folders listed in source.<library> according to the projects Java Build Path settings. I notice that a couple of the jdt plugins that I looked at (core & ui) do not have these properties set. Note that these properties are also used when generating the classpath. Not having these set may prevent other plugins depending on core & ui from being exported without having core & ui exported at the same time. As an aside, see also bug 88524 which shaved about 30 seconds of the export of jdt.ui in my test.
There is also some question about what to do with the customBuildCallbacks targets, there are 3 in particular: pre.<library>, post.compile.<library> and post.<library> that perhaps should not be run in this case (?)
>I notice that a couple of the jdt plugins that I looked at (core & ui) do not >have these properties set. I assume there are many other plug-ins like this. If this is important then PDE should probably generate a warning.
(In reply to comment #21) > >I notice that a couple of the jdt plugins that I looked at (core & ui) do not > >have these properties set. > I assume there are many other plug-ins like this. If this is important then PDE > should probably generate a warning. The PDE team discussed this some more and we have concluded that the output properties are not required by PDE build in general (unless a project has no source folders to compile - say, like an orbit bundle that is pre-compiled). However, we need to provide PDE build with this information in order to export source bundles from the workspace. Rather then have projects maintain this information PDE UI will infer/generate the properties and pass them to PDE build (based on a project's source folders and output locations). This will avoid duplicating information in the build.properties that is already specified in the .classpath.
Created attachment 118687 [details] Patch for UI side This patch generates the info we expect PDE Build to need and passes it to the build generator using the PDEUIStateWrapper. The map we provide is of the form: String Symbolic Name > Map of output folders. The output folder map is of the form: String Library Name > Set of IPath output folders
Created attachment 118688 [details] mylyn/context/zip
Created attachment 118729 [details] updated patch This is an update to Curtis' patch to include implementation from pde.build. Curtis: I fixed an ArrayIndexOutOfBounds exception in #getPluginOutputFolders where you had sourceFolders[i] instead of sourceFolders[j]. Also note my changes to #setupGenerator which need to be made real.
Looking good.
Is everything OK when "Build automatically" is disabled?
Andrew, when would the PDE Build changes go in?
I have released the changes to build (with small differences from the patch). This must be turned on by UI before it will get used. Workspace class files will get used if: 1) UI calls BuildScriptGenerator#setUseWorkspaceBinaries(true) 2) UI provides the src->output map via setStateExtraData 3) The provided map contains entries for the source folders in question
Working on this now. Basic option to enable the feature is ready to go, but we are looking into building and checking for errors before exporting.
Created attachment 119441 [details] Work In Progress Adds option to turn on the feature to UI (on the options tab of the export wizard). Tries to determine what projects are required for the export and builds them first and checks for errors.
Created attachment 119442 [details] mylyn/context/zip
Created attachment 119551 [details] PDE UI Patch Adds the option to the UI side of things, does an incremental build on required projects and tests for errors in the workspace. The code is not committed yet because in my simple test case (debug core and debug ui) I did not see any significant performance gain when turning on the option. In fact, if the workspace wasn't already built, I saw a significant performance hit. I don't know if I am failing to set a required option or the map does not contain the correct information. Timing the build script execution did show some improvements, but they varied significantly (5%-50% reduction). We should do some more testing before committing this code.
We must copy the class files from the binary folders to the export staging area. When not using this option, we compile files directly to that location. So any performance gains come from the difference between the time it takes to copy and the time it takes to compile. If the bundle was not previously compiled then you are right and we would actually expect a performance hit since in that case all we have really done is add an extra copy to the whole process.
Curtis, how about adding a preference to the plug-in development preference page so it applies to product exports too? The preference would be off by default.
> In fact, if the workspace wasn't already built, I saw a significant > performance hit. I don't think that happens often in practice. Before you export from your workspace, you typically want to rule out compile errors and run the automated tests.
Committed the PDE UI patch to HEAD.
Done in M4.
I gave this a try and it takes about 90 seconds to export 'org.eclipse.jdt.ui' using existing class files and also 90 seconds without using existing bits :-( Does it really use the existing bits? If so, why does it take the same amount of time? Maybe the copy code is very slow? If I use our ant build script it takes 6 seconds! So, while the functionality to use the class files seems to be here now, it doesn't really fix our problem.
(In reply to comment #39) > I gave this a try and it takes about 90 seconds to export 'org.eclipse.jdt.ui' > using existing class files and also 90 seconds without using existing bits :-( Yes - from previous comments you can see that we found the same thing. It appears that copying files from the output location to a temporary packaging location is almost as expensive as compiling files (I do recall that I/O is one of the copmiler's bottlenecks). However, getting PDE to avoid this copy to a common area is less trivial work. I will open a new feature request for this to keep track of the item.
