Community
Participate
Working Groups
N20050323-0010 After importing jar'd projects into my workspace as binary projects I noticed that the import now creates a class folder which contains every single class file separately instead of one single jar. This has the following disadvantages: - larger footprint on disk - slower startup times of the target workspace especially if you have a virus scanner activated. - This adds significantly to the memory footprint of the development workspace. In the old solution we had the jar files in the resource tree. Now we have every single class file in the resource tree.
You are importing a plugin with a . on its classpath, and you are getting a project with classes at its root. The project compiles and runs fine. PDE is the middle man and can't do anything about it If you think this behavior is unacceptable, then this is philosophical discussion about JARd plugins and their adoption should be conducted with the runtime team.
The jar'd plugin approach moves us closer to the standard Java way of doing things. This is one of the benefits. The behaviour you are seeing is (I believe) the same as if you imported a standard Java jar into your workspace as a project (assuming that you can do that). We should step back from this a little and look at why you are importing as a binary project. I'm guessing that standard Java developers generally do not import normal jars as binary projects . The historical driver was simplifying plugin classpath management. PDE's classpath containers etc have pretty much solved that problem for some time now. Other reasons that have come up relate to search, browsing and source lookup. I believe that several of these are still issues. As far as I can see, these problems relate to jars in general, and are not specific to plugins. Can you describe your workflow and use of binary import a little more so we can see if there are point or incremental solutions that improve on the points you made?
I am a bit nervious about the implications of this. It puts pressure on PDE to solve some long-standing problems in terms of external plug-ins management (which is not bad in its own right). However I would like people who use binary projects to move to external plug-ins because there is nothing to loose in the move, not because they are forced to. With the plug-ins in JARs, we are fundamentally changing the self-hosting landscape. Not only that binary projects are becoming very hard to create, but also importing with links is impossible. I don't see a way to restore a more- less familiar binary project option with JARd plug-ins without writing additional PDE code in the sensitive area of plug-in import. I am not saying it is impossible, just forces our hand fairly late in the release.
I was actually focussing more on the Java side of the world. Plugins are now just jars like any other Jar. If issues related to searching, browsing, ... external Jars are resolved then they should be resolved for PDE as well.
The reason why I import the plug-ins as binary project is easy: I want to have a sandbox which is not affected by any changes I do to my developement environment. ZRH does a plug-in export every day and we use it for self hosting. So my development environment changes on a daily basis. And this shouldn't have any impact on the content of my workspace. The basis for the decision when to reference a jar and when to import it into my workspace depends on how frequent it changes. So I wouldn't import rt.jar but I want to import plug-ins to make sure that I can work in a sand box even if I change my development environment.
Great usecase. Would platform targets help you here? For example, I never target my IDE because, like you, want control over the sand box. Here sandbox = workspace + target. In my case it is forced by RCP constraints etc. but the issues are similar. When I start from scratch I get an I build for my IDE (c:\ide) and whatever I want for the target c:\target (typically the RCP SDC). I start the ide and set the target to be c:\target. They then checkout only the projects on which I want to work. When I run I'm running the target install. For you, you might get the full SDK and unzip twice, once for the target and once for the IDE. This approach gives you complete freedom to change the ide independent of the target. Further, say you want to go to the next I build (assuming you currently sync up your workspace periodically), just delete c:\target and replace it. No need to reimport the binary projects. Does this make any sense or is it off in a different direction?
I also prefer to self host by importing the binary projects due to restrictions that Jeff already pointed out in comment 2. It is a something that worked for us and most likely for many others for a long time.
To be clear, it continues to work though with some issues as Dirk points out. I'm really not suggesting that people change the way they work but more trying to understand how they work and why. I agree totally with the comments from all of the D*s. Hopefully between JDT, PDE and runtime we can enumerate the issues and potential solutions. Wassim in particular has been working very hard to make the PDE tooling all that it can be and is very sensitive to people's development needs. But there is only finite time. Creative solutions in the JDT (UI and debug) space for dealing with external jars in general would likely help reach Dejan's goal of seamless migration and the other D*'s goal of equivalent behaviour.
Regarding comment #6: the setup makes sense, however it feels more complicated then what we do today. Today I update by eclipse installation with our daily export run it for a while and if it works I simply import the plug-ins form my installation. With an external approach I have to do it twice. Additionally disadvantages of this approach are: - no stable project layout when you switch between source and binary projects (which I do at least once a week). Currently, both binary and source projects appear as normal projects. Using the PDE class path container to reference external Jars I always have to think about if I have a project in source or in binary to look into the right container. - there are still problems how PDE manages the class path container (a good summary is in bug 79966) and I don't see an easy solution for this. I took a look at the layout of the jar'd plug-in and one solution would be to change the way how PDE imports binary projects. I will outline what I have in mind: - PDE creates a project with the name of the jar'd plug-in - instead of expanding the jar it simply imports the jar and puts it onto the class path and attaches the source file. - it copies the plugin.xml file out of the Jar and patches the library statement (e.g change library name="." to library name="`plugin_name`.jar".
Dejan and I were discussing this possibility. Although it would compromise the integrity of the plugin itself as we would be butchering its plugin.xml/manifest.mf, we were contempating doing it for the sake of uninterrupted workflows. However, a problem remains is that given a JARd plugin, all classes/resources are at the root of the JAR. PDE has no clue how to reverse-engineer it: ie what is supposed to be extracted to the root of the project and what is supposed to stay in the JAR. https://bugs.eclipse.org/bugs/show_bug.cgi?id=88559
Some thoughts: - We can contrive to have the export problem just take the plugin jar that is in the project. This would likely need some special markup but say bin.includes=@org.eclipse.foo_3.1.0.jar where the @means that this the thing. (period). the bin.includes would not have any other elements (or they would be ignored). - Tom Watson brought up another point, what about launching. When you run such a plugin you actually have to install the jar, not the dir. Hacking the bundle classpath in the manifest in the dir is enough to make the compiler happy but if you install the project dir, what about the icons etc. that are expected to be files in the root of the plugin? These are not accessed via getResource but rather getEntry - So, why not go whole hog and just leave the plugin as is, cereate a project, add the jar to the project as a lib and have special cases for exporting and running. In both cases you effectively ignore that it is a dir and install (in the case of running) or copy (in teh case of building) the pluigin jar itself.
Any suggestion that does not involve the end result being a workspace project with manifest files and other resources at the root of the project is a no go. Therefore, we cannot make progress on this issue unless the JARd plugin contains a file which PDE can read to deduce what should be at the root of the project and what should remain in a JAR (or unzipped into a source folder - bug 88559). I still have not heard a valid reason why supplying such a file (build.properties (full or trimmed-down) or the like) in the JARd plugin is not desirable.
I was not involved in the spec of build.properties, but can someone with more knowledge one this explain why this was mapped to a '.' instead of simply adding a 'jared' attribute? In my opinion this would have allowed to provide this featue and allow PDE tooling to do a good job. Please correct me if I'm on the wron path.
Dani, not sure what part of build.propreties you are refering to but in general there are things like source.foo.jar=<source dir locations> source..=<source dir locations> Here the thing after the "source." is the classpath entry for which the following directories hold the source. There can be many of these. For jar'd plugins their actual classpath entry typically really is '.'. For bin.includes you previously put in there the name of the jar(s) and other things. So, using the same rule ("replace your jar name with ."), the bin.includes entry becomes '.'. To be clear, the . doesn't really mean . in the filesystem case. It is effectively the same as your suggestion to use a special key like 'jared'. Anyway, to comment 12. yes, I was wondering about problems if the manifests were not in the root of the project as normal. When you say "resources", what are you talking about? Things like icons? I guess you need these for validation in the extensions etc right? Having the build.properties in there is possible. Question: What we really need is to distinguish between code/resources and other stuff right? So we are interested in the bin.includes line right? Could we use the Export-Package list to determine the answer? In the case that we are doing the classic export *, this list identifies all of the code in the plugin. Anything not covered in the package list would be extracted and placed in the project. The plugin Jar would remain attached to the project as normal etc. The issues come up when we go to run or export. In both of those cases we really want to use the jar not the project. A thought here is to "know" that the project is one of these binary imports of a jar'd plugin and so when building the list of plugins to run (in the platform.xml or config.ini) or export (however that is listed), list the plugin JAR rather than the dir. So the files in the project are just there to keep the tooling happy but otherwise the jar is the source for running/exporting. Does any of that make sense? None of the approaches is particularly appealling. For 3.1 we should look for something that enables the usecases put forward but does not imply yet more heroic effort from the PDE team.
I was thinking along Wassim's comment 12 where the build.properties file would be distributed. If it would also contain the old (unjared) information about what goes into JARs then PDE could import the project.
>I was wondering about problems if the manifests were not in the root of the >project as normal. This definition of a workspace plug-in being a project with a manifest at its root must remain intact. Otherwise, we will have chaos and disasters, and I'm talking like those biblical disasters that only locusts survive :-). First, more than 60% would have to be rewritten. Second, would PDE have to scan all JARs in the workspace to see which are plugins and which are not. >A thought here is to "know" that the project is one of these binary imports >of a jar'd plugin and so when building the list of plugins to run (in the >platform.xml or config.ini) or export (however that is listed), list the >plugin JAR rather than the dir. Third, keep in mind that binary projects are good in that they are exempt from JDT and PDE builds. however, they are still writable projects (if they are not linked). So users can add and modify plugin.xml files if they want to. You don't want to get into a situation where the user starts modifying the plugin.xml and yet when he launches we pass the JARd plugin instead. We cannot bend any of these basic self-hosting rules. >When you say "resources", what are you talking about? Things like icons? I >guess you need these for validation in the extensions etc right? Yes, and we have multiple uses for them. Most importantly, when you launch we pass the URL of the project to the runtime so these resources have to be at the root. >So we are interested in the bin.includes line right? Yes. >Could we use the Export-Package list to determine the answer? Not bad, but no. What if a plug-in does not export any packages. Also note that for plugins that have no manifest.mf, we do NOT analyze the jars as it is a huge overhead, so no export-package header is generated. We need to have closure on this in the next week or else it would have to be deferred until 3.2 Here is the only feasible solution: 1. the JARd plugin contains some kind of marker file enumerating the (non-java) files and folders that must go at the root of the plugin. 2. When PDE imports the plugin as binary, it extracts the list of files and folders from 1. at the root of the project and imports the JAR as-is. 3. Since the code is now in a JAR, the '.' on the classpath is no longer correct. Therefore, PDE will munge the plugin.xml/manifest.mf to make that change. (This is the part that I hate doing the most) 4. The End. Note that with this solution, the import with source problem would be solved as well. In that case, we wouldn't munge the plugin.xml, the source would go under a src/ folder and the build.properties would be complete.
You've got an answer for everything don't you ;-) Ok, it seems we have exhausted the options for the 3.1 timeframe and a flavour of what you suggest is what we have to do. The thing I'm trying to avoid is having an extra file and some magic to create/maintain that file. Let's go back to >>Could we use the Export-Package list to determine the answer? >Not bad, but no. What if a plug-in does not export any packages. Also note >that for plugins that have no manifest.mf, we do NOT analyze the jars as it is >a huge overhead, so no export-package header is generated. comment 16 suggested listing all the things in the plugin that are not code. Since there are only two kinds of things, code and not code, listing the code implicitly lists the not code. (I just had to write that out). The points you made against using the Export-Package list: - what if no exports: If there really are no exports then everytihng in the plugin would be, by definition, not code and would be listed explicitly on the bin.includes line. not code - no code = not code. - the issue about no manifest.mf is harder. In this case I suggest that if someone is importing such a plugin, we actually do the analysis (PluginConverter) and thus generate an Export-Package list. The downside here is that pragmatically, a jar'd plugin with a plugin.xml will have . on the class path and export=*. This means that in the end, the whole plugin will be exported and everything treated as code. Under the current model, nothing will be imported. I propose that we treat this as a pathological case and import the whole jar. I.e., when in doubt, import it all. The net of this approach is that none of the build process has to change. People do not have to release their dev-time files (build.properties) in the runtime, and we solve the main issues we have. We could always have a backup mechanism for the pathological cases which is somewhat like you describe but it would be good not to require that everywhere.
So. Is this going to be addressed for 3.1 ? JDT/Core is also using binary plugins (not linked) for the same reason Dirk mentionned. I also feel that asking clients to change the way of working is as good as simply stating this scenario is no longer allowed, which would be bad. The current state of imported binary plugins is simply overkill, especially considering the performance effort we are putting in for M7.
Side-note: one direct consequence of this evolution is bug 89848. Though we were able to address it, there may be more issues with this new setup, which has been little tested before (as proved by existence of bug 89848); and we are getting close to 3.1... so why changing rules right before a release ? We now that imported binary plugins as JARs are working just fine.
philippe, I'm not sure what you are suggesting here. Do you object to the proposals in comment 16 or comment 17? If so, how and do you have a proposed alternative?
and now for the rebuttal :-) >what if no exports: If there really are no exports then everytihng in the >plugin would be, by definition, not code and would be listed explicitly on >the bin.includes line. not code - no code = not code. Export-Package is not sufficient. What if a JAR hypothetically has two sets of packages: ones that start with com.example.bla.* and others that start with private.monkey.* and the Export- Package only want to expose the former. All the private monkey classes will end up at the root of the plugin. >the issue about no manifest.mf is harder. In this case I suggest that if >someone is importing such a plugin, we actually do the analysis >(PluginConverter) and thus generate an Export-Package list. Doing this analysis takes a couple of minutes for a target platform with 1000 plugins. So you will have to wait a couple of minutes when you want to proceed from the first to the second page. >a jar'd plugin with a plugin.xml will >have . on the class path and export=*. This means that in the end, the whole >plugin will be exported and everything treated as code. Under the current >model, nothing will be imported. I propose that we treat this as a >pathological case and import the whole jar. I.e., when in doubt, import it >all. That is what all the current JARd plugins that have a plugin.xml in the SDK look like. >The net of this approach is that none of the build process has to change. >People do not have to release their dev-time files (build.properties) in the >runtime, and we solve the main issues we have. Time to add the build.properties to the bin.includes, save the file and release it into HEAD: 2 seconds. Time to come up with alternate solutions: 8 days and counting :-) When importing the content of the jar, I am not going to import the build.properties as-is, I will use it as a guideline and create a trimmed down version.
Re: comment#20 Reading through the PR, it wasn't clear to me whether this would be addressed for 3.1, or rather deferred. If resolution is for 3.1, then I am fine with the current direction.
Moving to PDE/UI to resolve this issue. In return, I get custody of Jeff's boat for the month of August.
sure if you want to come up and do all the spring maintenance/cleaning...
Thanks !!
Oops I thought it got fixed, but 92550 got fixed. But given this is fix isn't far away...
Done. Will appear in the next nightly build. When importing a JARd plugin, we will: 1. extract the non-Java resources to the root of the plugin 2. import the JARd plugin as-is into the project. 3. modify the manifest.mf of the project to reference JAR from 2 not '.' In the case of a linked import, in step 2, we will link to the external JAR. Importing as source is also back to normal now. We will extract the java files into a source folder, not at the root of the project. In the case of import with source, we don't modify the manifest.mf. We just map the '.' to the source folder in the build.properties file. This effort took an insane amount of work. It's a testament to show how far a guy would go for a boat ride :-) I should be eligible to become a pope now. I pick the name Pius.
Cudos for Pius!
What plug-ins do I need from HEAD to get this? I just took org.eclipse.pde.ui from HEAD and it imported each class file so I guess I also need HEAD of some other plug-in(s). And of course Kudos, not cudos!
it's still in a branch (Binary_Import). You would need pde.core and pde.ui. will release into HEAD later on today.
You can definitely steer the boat. We'll even set up the BBQ!
