Community
Participate
Working Groups
I'm not sure if this bug is best placed in equinox or in platform, feel free to reassign. This change was probably introduced for implementing the HTTP service but this can cause problems for other plugins. Particulary I'm embedding Jetty as webcontainer, another version of MX4J jmx management, I have my own javax.servlet bundle, jasper bundle, org.apache.commons bundles etc .. It seems strange to me that the tomcat plugin should export so many packages. On top of this the tomcat version delivered is outdated and includes outdated components. I know I can solve this using OSGi conflict resolution versioning rules and I now in the process of changing all my bundles, but I'm guessing this will affect a lot of other plugins as well. Even if the tomcat bundle needs to export all those packages, it would be far better if these would be split in seperate bundles.
tomcat is owned by user assistance
The packages are marked as internal, so they are not exposed for consumption. I'm not sure how this is causing conflicts.. This also conflicts with the 3.2 checklist for plugins that go in the SDK (sent to eclipse-dev): ---- Below is a checklist that plug-ins must pass to get a clean bill of health for the 3.2 release. I know it's only M5, but with the simultaneous release of many Eclipse projects at the end of June along with their translation, it's never too early to address these issues. The good news is that, as of 3.2M5 (I-20050215-1600 to be exact), PDE provides tooling to help you address all these issues in less than 5 seconds. 1. Exported Packages: - All plug-ins in the SDK must enumerate all their packages in the manifest.mf - API packages and internal packages must all be listed with the appropriate visibility manifest markup. ...
Curtis, it is possible that we are exporting internal packages and marking them as internal, which is fine from the tooling point of view (references will be flagged) but nevertherless puts these classes on the classpath that can clash with conflicting packages. Added Wassim to elighten us on this topic.
This is not an x-internal vs. unconditionally exported issue/discussion, since we never run in strict mode. This problem is, I hear, due to the use of buddy classloading in Help which is causing problems. I know the runtime team discourages the use of buddy classloading unless it's an extreme case. I recommend reverting the manifests to the M4 state and seeking the advice of Tom or Jeff to resolve the original issue for which buddy classloading was initially introduced.
I have rolled back the addition of the buddy policy for bug 113252 so this bug should no longer happen. We will have to find another solution for that bug. Tom or Jeff, can you comment on how we could resolve bug 113252?
As a point of interest, what other things is the tomcat plugin exporting now? There were other issues around the fact that it was exporting the servlet api. I propose that we remove the internal exports for all tomcat packages in the org.eclipse.tomcat plugin. Reopening for consideration.
Just to clarify. Are you suggesting we remove just the x-internal statement from each of the relevant package exports or removing the package exports altogether? (I'm hoping for the latter but want to be sure)
yes, the latter.
It currently exports: Export-Package: javax.management;x-internal:=true, javax.management.loading;x-internal:=true, javax.management.modelmbean;x-internal:=true, javax.management.monitor;x-internal:=true, javax.management.openmbean;x-internal:=true, javax.management.relation;x-internal:=true, javax.management.timer;x-internal:=true, javax.servlet;x-internal:=true, javax.servlet.http;x-internal:=true, javax.servlet.jsp;x-internal:=true, javax.servlet.jsp.tagext;x-internal:=true, mx4j;x-internal:=true, mx4j.loading;x-internal:=true, mx4j.log;x-internal:=true, mx4j.persist;x-internal:=true, mx4j.server;x-internal:=true, mx4j.server.interceptor;x-internal:=true, mx4j.timer;x-internal:=true, mx4j.util;x-internal:=true, org.apache.catalina;x-internal:=true, org.apache.catalina.connector;x-internal:=true, org.apache.catalina.connector.http;x-internal:=true, org.apache.catalina.connector.http10;x-internal:=true, org.apache.catalina.realm;x-internal:=true, org.apache.catalina.servlets;x-internal:=true, org.apache.commons.beanutils;x-internal:=true, org.apache.commons.beanutils.converters;x-internal:=true, org.apache.commons.beanutils.locale;x-internal:=true, org.apache.commons.beanutils.locale.converters;x-internal:=true, org.apache.commons.collections;x-internal:=true, org.apache.commons.collections.comparators;x-internal:=true, org.apache.commons.collections.iterators;x-internal:=true, org.apache.commons.digester;x-internal:=true, org.apache.commons.digester.rss;x-internal:=true, org.apache.commons.digester.xmlrules;x-internal:=true, org.apache.commons.logging;x-internal:=true, org.apache.commons.logging.impl;x-internal:=true, org.apache.commons.modeler;x-internal:=true, org.apache.commons.modeler.ant;x-internal:=true, org.apache.commons.modeler.mbeans;x-internal:=true, org.apache.commons.modeler.modules;x-internal:=true, org.apache.commons.modeler.util;x-internal:=true, org.apache.coyote;x-internal:=true, org.apache.coyote.http11;x-internal:=true, org.apache.coyote.http11.filters;x-internal:=true, org.apache.coyote.memory;x-internal:=true, org.apache.coyote.tomcat4;x-internal:=true, org.apache.jasper;x-internal:=true, org.apache.jasper.compiler;x-internal:=true, org.apache.jasper.logging;x-internal:=true, org.apache.jasper.runtime;x-internal:=true, org.apache.jasper.servlet;x-internal:=true, org.apache.jasper.util;x-internal:=true, org.apache.jasper.xmlparser;x-internal:=true, org.apache.naming;x-internal:=true, org.apache.naming.factory;x-internal:=true, org.apache.naming.java;x-internal:=true, org.apache.naming.resources;x-internal:=true, org.apache.regexp;x-internal:=true, org.apache.tomcat.util;x-internal:=true, org.apache.tomcat.util.buf;x-internal:=true, org.apache.tomcat.util.collections;x-internal:=true, org.apache.tomcat.util.compat;x-internal:=true, org.apache.tomcat.util.handler;x-internal:=true, org.apache.tomcat.util.http;x-internal:=true, org.apache.tomcat.util.http.mapper;x-internal:=true, org.apache.tomcat.util.log;x-internal:=true, org.apache.tomcat.util.mx;x-internal:=true, org.apache.tomcat.util.net;x-internal:=true, org.apache.tomcat.util.net.jsse;x-internal:=true, org.apache.tomcat.util.res;x-internal:=true, org.apache.tomcat.util.threads;x-internal:=true, org.eclipse.tomcat.internal;x-internal:=true, org.eclipse.tomcat.internal.extensions;x-internal:=true What are the implications of removing these? I assume there was a reason for requiring all plugins to enumerate all their packages.
Yes, there was a reason and lest anyone accuse me of hypocracy (Wassim!), the point was to facilitate the community working with Eclipse code that was not fully baked (e.g., provisional API) and hacking into internals if they absolutely needed to (e.g., to ship their product or pursue innovations). These reasons are all entirely valid. The case with Tomcat is somewhat different in that a) we don't produce this code b) there is a large collection of apache and other stuff that its almost entirely implementation detail related to parts of Tomcat that are not even used by Help c) there are conflicts with parts bits that other people need/want to use in other contexts (e.g., servlet api, JMX) d) The use of Tomcat is entirely internal implementation detail. So, it is somewhat spurious that we expose the packages.
Here is the official announcement with respect to exporting all packages: http://dev.eclipse.org/mhonarc/lists/eclipse-dev/msg07092.html People at the time seemed to have been concerned mainly about packages in the Eclipse code itself, and not about 3rd party plugins like lucene or tomcat. The announcement leaves room for exceptions. Tomcat seems like a reasonable one. In any event, being this late in the cycle, it would be prudent to advertise on mailing lists the fact that we will be hiding internal tomcat packages, and see if anybody raises a valid objection.
Unfortunately some of the org.eclipse.help.tests use classes from, for example, catalina. I need to investigate these..
This is not unusual. You can use several approaches to resolve this. One is to have a fragment to Tomcat that adds x-friend exports for the relevant packages.
The Buddy-Policy may have been related however in M6 all of the resolve problems that Erik mentioned are still present. This makes it quite akward to use Eclipse as an OSGi dev platform when any of the dependencies are involved. Is the Export list likely to get cleaned up for 3.2 or is it too risky now? Is there any way that I could help here?
I would like to try and get a sense of how severe the problem is. Can you help me understand, briefly, what you can't do with these exports present, and how bad the workaround is?
Hi Curtis, What you can't do easily (with these exports present) is define plugins that export or import the same packages from a different provider. For example, even if you define a plugin in your workspace that exports apache.commons.logging, there's no guarantee that if you define another plugin that imports ACL you won't end up instead resolving against the tomcat plugin and getting a number of "discouraged access" warnings. The easiest workaround is for this sort of problem is probably to add additional "version" information to the exports on your ACL bundle so that it will take precedence over the non-versioned export in the tomcat bundle. -- Another example... For the OSGi Http Service the problem is a little bit worse because the org.eclipse.osgi.services bundle will resolve and bind against javax.servlet and javax.servlet.http packages from the tomcat bundle at RESOLVE time. This is particularly problematic because then even if you also have a versioned Servlet API bundle in your workspace it can not be used for resolving javax.servlet imports if the bundles also imports the OSGi Http Service. {Sorry -- hopefully that makes sense} The only way I've been able to workaround this problem is by defining a different "target platform" that excludes the tomcat bundle or in some cases I've modified the default platform to include the updated Servlet API and other packages. -- I've just begun to explore some of the Jasper/JSP issues but expect more of the same as I start to need specific versions.
If you want to take a first hand look at some of these dependency problems for yourself take a look at bug 132555. It's looking to embed Jetty as well and brings in specific requirements for a number of the packages also exported by the tomcat bundle.
Tentatively targetting RC2. Will need to do some testing to ensure that removing the exports won't have any negative consequences.
Mind you that there are a lot of bugs open which end up being this problem (log4jlogger does not implement log issues) so several people ended up reporting this. I'll give another example as well, one that actually has no workarround that I see. We use jmx, and the tomcat plugin contains a fairly outdated version of mx4j. We want our set of plugins to work with jdk 1.4.2 and upwards so we have to use an Import-Package statement which will resolve against the javax.management packages in rt.jar for jdk 1.5 and 1.6 but against our own mx4j plugin in jdk 1.4.2. Sometimes it however resolves against the tomcat plugin causing dreaded ClassNotFound Issues or the likes. In this case I cannot place a version restriction arround it, as I have no control what the system bundle exports. Another example is apache regexp. The regexp included in tomcat is another and API incompatible version than is included in Xalan XSLTC compiler. Xalan is often placed in the endorsed directory, causing also problems. As I mentioned before some of our developers have experienced what others seem to have reported, namely the commons logging issues not allowing them to open up the help at ALL. I've gone so far as to include special lines in our ant script as to completely ignore the tomcat plugin during ant builds and hope that at runtime it works. Unfortunately as we have seen here depending on the order (or some other magic we haven't figured out) in which plugins were installed in the framework the resolving may go bad. I think these are all serious problems. It has been stated in numerous bugs that tomcat is just an articfact of how the help system is implemented and may change . Exporting the packages in this case is bad. Suppose you are going to switch to jetty, people will rely on the packages exported by the tomcat plugin and all of a sudden half of these will be gone (jetty has far less dependencies). Same if you switch to a newer version of tomcat. If you want to export these then these jar files needs to be proper plugins with versioning. I admit I don't understand why not exporting these packages proves to be a problem. Is it simply because of the test plugin mentioned above ? If not can someone give a short explanation. I have had my fair share of eclipse OSGi experience the last few months so maybe I can suggest a solution ?
The exports are remove in build I20060412-1200, which is now available. Erik and Simon, could you confirm this fixes the problems?
Thanks Curtis. I can confirm all of the problems are gone. I'll keep trying over the next few days but looks good to me. -- Aside... Even the "Add Required Plug-ins" button in Run/Debug is working. I haven't had that working for months (maybe even a year ;)
May I reopen, requesting clarification on why this change was made (especially so late). Sorry if I'm dense, but I couldn't really find the problem what was being solved ... seemed a little like "clean up" to avoid some theoreticial problems? We in WTP have several uses where we expect some of these packages to be "visible" when we build WTP, and at runtime, we expect users to have "real" servers available to provide implementations especially for things like TagExtraInfo. Even at runtime, our code still works well, since if no "real" servers defined, we get back null at all the right places, and just don't execute. Seems like now we'll get a "class not found" error in code that refers to those interfaces. This particular problem could perhaps be "fixed" by listing a small set that Nitin posted on eclipse.dev mailing list? But, I believe Web Services "Explorer" still expects it to be there, just like the help system does, to launch a web app. I'm adding Chris to CC list to see if he has been aware of this and already fixed/migrated ... perhaps my information is old. And ... before it starts ... please don't reply with "this is internal, should not be used" ... there's a long, long history of discussions with previous owners boiling down to "yeah, not great, but no reason its going anytime soon, so, future problem". So ... first, 1,001 thanks to Curtis for posting a warning to eclipse.dev list! and second, Curtis, is your offer to "help migrate" still applicable to the types of uses I list above?
the Servlet API are availalble (and versioned) in the org.eclipse.equinox.servlet.api bundle available from the Equinox download site http://download.eclipse.org/eclipse/equinox
Downloading integration build as we speak, will test before the end of the day and post my results. (In reply to comment #22) > May I reopen, requesting clarification on why this change was made (especially > so late). Sorry if I'm dense, but I couldn't really find the problem what was > being solved ... seemed a little like "clean up" to avoid some theoreticial > problems? I would not have openend this bug had I not experienced real classloading and classcast problems ;-) So have several others in my company. In my opinion this should also fix the bugs which are opened against help by several sources of the Log4JLogger does not implement Log. Personally my biggest gripe is with anything that is exported that may end up in the bootclasspath of the jvm, and thus end up being exported by the system bundle. At that point you can't define strict enough Import rules to work both when they are in the bootclasspath and when not. The most likely candidates are mx4j's javax.management classes and anything commonly included in endorsed directories. As for the theoretical part this is exactly what I was afraid of, so WTP does not ship its versions of javax.servlet and such but instead relies on the unsupported and very outdated versions in the tomcat help ? What if a customer actually inserts a tomcat or some other webcontainer with conflicting packages, are your import rules good enough to have the OSGi kernel do the conflict resolution ? How does WTP deal with that ? Why wasn't this a problem in previous releases of WTP for 3.1.x ?
(In reply to comment #24) > > Personally my biggest gripe is with anything that is exported that may end up > in the bootclasspath of the jvm, and thus end up being exported by the system > bundle. Thanks for clarifying ... who invented that bootclasspath and endorsed directory stuff anyway :) ... that pretty much will always conflict with OSGi design goals right? Short circuiting OSGI's ability to load and find classes "smartly"? > Why wasn't this a problem in > previous releases of WTP for 3.1.x ? > I think only because the few cases we rely on are simple interfaces that haven't changed ... and ... no one I know puts javax.servlet stuff on the bootclasspath anyway (or, would want to ... that I know of). BTW, WTP is actually chartered to NOT ship servers ... even though users install a variety of them, and we adapt to what's installed. So ... we may have to approach the Eclipse Foundation and submit paper work to re-distribute sun's servlet API's? Well, that's rhetorical, not a question for this bug. The use of the web services explorer "web app" is more straight forward ... its doing exatly the same thing help is doing ... just launching a simple web app. And, as I see it, that's the harder problem to solve here (unless my information is out of date, and that team has already solved this). FWIW, still sounds like throwing the baby out with the bath water :) But, I do better understand the motivation now, thanks again.
David, it sounds like you not only need the classes to be available at build time, but they also need to be shipped in order to be available at runtime - can you confirm? If this is the case, would it be possible for you to ship the equinox bundle Jeff mentioned as part of WTP? Also, for the web service explorer, were you using the org.eclipse.help.appserver internal code or using the org.eclipse.tomcat directly? If the former you should be ok, as long as you're not directly using org.eclipse.tomcat.
FYI, we are still investigating in WTP, probably won't have firm conclusion until early next week, but will investigate making use of the equinox bundle for serlet APIs. I'm 80% confident the conclusion will be, a. we do use org.eclipse.help.appserver to launch the web app, so may be ok. (In a class cleverly called CatalinaRunnable :) b. we will have to (re)ship org.eclipse.equinox.servlet.api b1. I think we'd be ok using it compile time only for jsp components, as its a narrow "taglib" function it gives us, and by design, we handle "class not found" at runtime, the 'help' plugin has always been optional (and if some server is present, we reflect off its classpath). (remember ... 80% sure :) b2. the "web app" use of servlet api appears not so isolated, from my quick skim reading of the code, so it will likely require shipping servlet api's. So, the point of this "staus report", feel free to (re)assign this to fixed status ... It is in your RC1, right? ... and, I'll open new bugs if we have further issues reacting to your change ... or, need that equinox bundle spruced up a little! (its about.html seem out of date). I think this solution (if indeed works out) will (still) solve the log4jlogger type problems discussed in some comments ... even if WTP installed ... but my head starts to spin around those HTTP Service comments :) So, again, many thanks for re-hashing for me what you all have discussing for the past month and making the concrete suggestions for migration.
Thanks David. Setting as fixed again.
FYI ... for those interested, I've opened bug 136954 to track the WTP specific issues. Thanks.
See also bug 136954 for my motivation in adding the "greatbug" nomination.
FYI, Due to the easter holiday it took me more time than expected, but I can confirm that both in the integration build and RC1 our classloading issues have been fixed. I have just moved all our developers to RC1 as PDE target. Many thanks to the people involved and I hope I did not cause to much sleepless nights for David and his peers from WTP.
