Community
Participate
Working Groups
Here's my scenario. I have 4 plugins: 1. Axis plugin, which contains no source code, only contains 3rd party jars. One of the jars is called jaxrpc.jar, which contains the javax.xml.transform.dom.DOMSource class file. Note that this file also exists in the JDK. 2. EMF-WSDL4J plugin, which has a jar called wsdl4j.jar. This jar is not exposed, ie, not part of the Bundle-ClassPath. This jar contains a class file called javax.wsdl.WSDLException. 3. WSDL4J plugin, which contains no source code, only contains 3rd party jars. One of the 3rd party jars is called wsdl4j.jar. This jar is exposed (part of the Bundle-ClassPath) and again it contains a class called javax.wsdl.WSDLException. 4. Web service plugin, which depends on all the plugins above, and uses both the javax.xml.transform.dom.DOMSource class and the javax.wsdl.WSDLException class. My observations: a) start with a brand new workspace b) change my target platform to one that contains all the plugins mention above c) check out the Web service plugin only from CVS d) I'll get access restriction errors regarding the two classes: DOMSource and WSDLException e) If I then check out the Axis, EMF-WSDL and WSDL plugin from CVS, then all errors go away. It looks like things start to act funny when a class is loadable from more than one location. Btw, if I run my target platform standalone (not run-time workbench), things still work.
Access restrictions take place when a plugin's dependency has a manifest.mf. If they don't, then PDE is too lenient and can't enforce anything. I am not sure what the bug here is or why it's major.
Thanks for the quick update. I've just converted all my plugin.xml to manifest.mf, and this error starts showing up. I believe this is a bug because: 1. Everything works at run time (with the manifest.mf files). 2. If I use org.eclipse.releng.basebuilder to build my plugin through Ant scripts, everything works fine (no compile errors). 3. In the above scenario, checking out the one Web service plugin from CVS will result in access restriction errors. These errors will actually go away simply by checking out the other three plugins (Axis, EMF-WSDL4J and WSDL4J) from CVS. But the issue is that these three plugins are already in my target platform and they are the same. So it should make no difference whether I check out these plugins or not. The bug here is that I should not see these access restriction at the first place when only the Web service plugin is checked out. I consider this problem major because it is not compiling my Java source files. I checked the class files, it replaced all the method implementations with: throw new Error("Unresolved compilation problems: \n\tAccess restriction:..."); Again, I want to emphasize that my Java source does compile: 1. when org.eclipse.releng.basebuild is used 2. when the three plugins (Axis, EMf-WSDL4J and WSDL4J) are checked out from CVS. Note that these three plugins in CVS are the same as the one from my target platform. Although there is a workaround to this problem (check out additional plugins from CVS), it is not obvious. For instance, when new developers comes along and want to submit a patch for some problem in the Web service plugin. They check out the Web service plugin and find that things do not compile. Since the workaround is not obvious, there's a chance that the developer will just abort and think that the code is not stable... This hinders community involvement. Either way, Java class not compiling in the workbench environment, whereas, the same Java class compiles in the org.eclipse.releng.basebuild environment is serious enough to merit a major.
You should not be seeing compiler errors if your target plugins have no manifest.mf. As I said for traditional plugins, PDE is overly forgiving and that is the problem that access restrictions can finally address. what Eclipse build are you using?
My target plugins have manifest.mf. We have converted all the plugin.xml to manifest.mf. I do understand that we might be taking advantage of the "overly forgiving" side of PDE, however, I think at the very least, we should get consistent behavior between PDE/org.eclipse.releng.basebuild/run time. Actually, a question that is not really related... but I'll ask anyways. It's JDT whose doing the actual compile, right? Does JDT ask PDE for the "classpath"? I'm using the Eclipse RC1 driver and my target environment is also based on RC1.
Are the manifests in the target identical to the ones you check out from CVS. I would find it surprising if they were, and yet you were getting different compiler behavior. Can you attach a manifest.mf from a target plugin that you claim is causing false errors in your workspace plugin? JDT compiler does ask PDE to resolve the classpath, yes.
Created attachment 22258 [details] manifest files Consider the following two errors: #1. Access restriction: The type DOMSource is not accessible due to restriction on required library D:\wtp_0601\eclipse\plugins\org.apache.axis11_1.1.0\lib\jaxrpc.jar #2. Access restriction: The type WSDLException is not accessible due to restriction on required library D:\wtp_0601\eclipse\plugins\org.apache.axis11_1.1.0\lib\wsdl4j.jar For #1, javax.xml.transform.dom.DOMSource is not visible because its package is not exported by the org.apache.axis11_1.1.0 plugin. However, javax.xml.transform.DOMSource is actually loadable from the JDK. So question is, why is this being flagged as an access restriction? For #2: javax.wsdl.WSDLFactory is not visiable because wsdl4j.jar which contains this class is not exported by the org.apache.axis11_1.1.0 plugin. However this class is actually loadable from another plugin which does export its package. And this other plugin is on the list of Require-Bundles. So again, why is this being flagged as a restriction?
Philippe, this is an interesting case where a class by the same qualified name is available from two different sources with different visibilities.
Since I have to check if I can download all plugins needed to reproduce this problem, and this can take days, it would be faster if you (Jeffrey or Wassim) provided simple plugins that showed the problem.
Created attachment 22419 [details] testcase.zip Hi, Here's a testcase. This zip file contains 2 plugins: a and b. Plugin a has a class call java.lang.String, but this class is not exposed. Plugin b depends on plugin a, and has a class that uses java.lang.String. The following errors shows up in the problems view: Access restriction: The type String is not accessible due to restriction on required project a Main.java String cannot be resolved Main.java
Thanks a lot. I was able to reproduce and I'm investigating.
We follow the order of classpath entries. Since the entry that defines a non-accessible rule for java.lang.String comes before the entry for rt.jar's java.lang.String, String becomes inaccessible. Changing the order of classpath entries (through the Build Path dialog) and putting the JRE Container before the PDE container fixes the problem. Did I miss anything ?
This would artificially fix the problem because the class in question is java.lang.String. The scope is broader than that. Check out case#2 from comment 6. In there, two types coming from different entries in the PDE container have different visibilities, and the first one, whichever may be, wins.
Indeed this is expected. What do you propose ? Should we look at all entries and take the lowest restriction ? Or the highest maybe ?
the lowest restriction is best and safest, since that's what the runtime ultimately does. The classloader goes through the classpath entries linearly until it hits the visible package or falls off the edge and reports a classloading error.
actually, I think the problem here may be more difficult for the compiler than it is for the runtime. The runtime is completely unaware of the invisible packages. The compiler on the other hand is. So the code is really compiling against the first class of a given qualified name in the linear classpath, no matter what visibility it has. Correct?
Exactly. If we continued to lookup types (even if we found one with a non-accessible access rule), we would break the Java spec (a reference compiler would never generate such bytecode). We might want to change the runtime behavior.
The runtime of course is completely out of the picture in this case. Unfortunately, I am not sure there is a solution to this problem.
I meant that the runtime might consider throwing a NoClassDefFoundError if a type is included in a plugin but not accessible.
Jeff - is the runtime behavior mandated by OSGi or purely custom ? The implementation treats restricted items as if they were missing; which can makes sense somehow (though no compiler would follow you there), question is: who enforces that behavior ?
In this case OSGi mandates the behavior. That is if I understand the scenario correctly. Here is a simple case: Bundle A Bundle-SymbolicName: a Export-Package: foo Bundle B Bundle-SymbolicName: b Export-Package: bar Bundle C Bundle-SymbolicName: c Require-Bundle: b,a In this case say both bundles "A" and "B" have a package "foo" but bundle "B" does not export "foo". Bundle "C" requires b,a. When Bundle "C" loads a class from package "foo" the runtime will see that bundle "B" is required but does not export "foo" so it continues to bundle "A" which does export foo so it searches "A" for a foo class. The runtime does not access the content of "B" to determine that it should not search "B" for the foo package. The fact that "B" does not export the package implies that the package is not available to load from other bundles. I think you are suggesting that the runtime should search bundle "B" for package foo even though it does not export it AND if it finds a result then throw a NoClassDefFound error. This seems wrong semantically (to me at least) and it will remove any classloader optimizations we get from not searching all required bundles for every class regardless of what packages they export. We can argue semantics but the classloader optimizations are probably to important loose.
I am not suggesting anything, only trying to understand the difference in semantics. Basically, on the compiler front, we cannot break language semantics either. Restrictions are not in the language, and using them we cannot make compilation more permissive than the JLS. The only way I could imagine reconciling the 2 stories is for PDE to use exclusion filters in combination with access rules. When excluding something from the classpath (in case it has a better match somewhere else further), the compiler would simply ignore it altogether, and thus it the next occurrence of a type on the classpath. But, this means if you do not find any, the compiler will treat the type as purely missing.
OK, "suggesting" is too strong of a word ;-) What you state in comment 21 seems like a more correct behavior. Either way the resulting class file would be unusable, right? throw new Error("Unresolved compilation problems: \n\tAccess restriction:..."); Is inserting the above statement into the class better than not allowing the class to compile because of a missing type? BTW, at runtime the missing type would really not be available anyway.
If I understand comment 21 correctly, then there is hope and it would be localized. For every classpath entry, the last access restriction rule is always an exclusion access rule with a **/* pattern. Philippe, how can I use an exclusion filter to substitute for that last rule?
well, we have to do something here for 3.1, and we have to do it without breaking the JDT compiler. From comment 6, there are two cases: 1. a restricted type has the same qualified name as a non-restricted type coming from the JRE. For this one, I suggest that PDE, from now on, puts the JRE container ahead of the PDE container when creating a new .classpath file (upon import, or new plugin project creation) and when people use the PDE Tools > Update Classpath context menu item. A user's first reaction when he gets unexpected compiler errors is to do a PDE Tools > Update Classpath. So this approach would certainly work. There is no need to do a full migration of all existing .classpath files to switch the JRE and PDE containers because this conflict case is relatively rare. This use case aside, the JRE container should be ahead of the PDE container in the .classpath file anyway, because the runtime delegates to the boot path classloader before the plugin's classloader when loading a class. Case #2: Two classes with the same name coming from two different plugins have different visibilities. With package naming conventions and distinct plugin ids, this scenario shouldn't really occur. But we can't force people to follow these guidelines and this scenario seemed to have occurred here. As for what to do, I'm still not sure what comment 21 exactly referred to. If there is a safe thing to be done at this point, I'm all for it. Otherwise, this sounds like a README. In any event, I think only PDE could do something here, and this bug should be moved back to the PDE/UI bucket. Philippe/Jeff, input?
I believe so. The idea in comment#21 is likely super expensive to implement. You would need to compute the intersection in between two libraries, and realize that for some of them, you simply want to exclude them from first entry, so that only the last occurrence is considered by compiler. Is that a realistic effort ? Could swapping entries solve the issue you are seeing (as you suggest for JRE case).
Actually, I spoke too fast. Exclusion rules (not restrictions) can only be specified on source entries... so comment 21 is not a valid option.
I think I would document this issue for 3.1; it is very specific to our platform runtime; where access restrictions means classfile filters (i.e. you cannot tell if classfile is present but hidden from being absent).
Thanks Philippe. I will move the defect back to PDE/UI to address the switching of PDE and JRE containers in the .classpath file. That's all that can be done.
If I understand correctly, case #1 can be workarounded by switching the order of the JRE container and the PDE container in the .classpath file. So what about #2? Is there a workaround for it? Does the PDE container go throught the classpath entries in the order that is defined in the manifest file? If so, does that mean putting the bundle with the visible class before the bundle with the restricted class will at least workaround problem #2? Thanks.
Yes, we navigate the dependencies in the order in which they appear in the manifest.mf.
Created attachment 23199 [details] Patch for pde.core
Created attachment 23200 [details] Patch for pde.ui These patches make sure that the JRE container entry appears before the PDE container. Konrad, please review for RC3.
The patches look OK for RC3.
thanks.
Since this bug is put into the resolved state, does that mean case #2 is working as is? If so, then we need a official statement on what should clients do when they have the same fully qualified classes with different visibiliities that are loadable from different sources. Can clients rely on the fact that the visibility is determined by what gets loaded first, and the loading order is determined by the order defined in the manifest file? Also, is this mechanism supported by Eclipse?
re comment 35, we are talking about compilation at development time, therefore I would stay away from the word 'load' and all its derivatives. Facts: 1. If the classpath contains more than one type by the same qualified name, your code compiles against the first type by such a name on the linear classpath. 2. When resolving the Plugin Dependencies container, it is the order of required plugins as they appear in the manifest.mf that dictates the order of JARs/projects in the resolved classpath of your plugin. If a plugin's classpath contains two classes by the same name with different visibilities from two different dependencies, the dependency that gives the greater visibility must appear first in the list of dependencies of the manifest file
Hi Wassim, thanks for the clarification.
Sorry to reopen old sores but this problem is popping up in other places that are pretty serious. Consider Bundle-SymbolicName: B Export-Package: foo; version="1.0",bar Bundle-SymbolicName: C Export-Package: foo; version="1.1" Bundle-SymbolicName: D Import-Package: foo; version="1.1",bar The state and PDE are correctly resolving D to get foo from C and bar from B. So far so good. The project for D has a PDE classpath container that lists B - Accessible: bar - Forbidden: **/* C - Accessible: foo - Forbidden: **/* This is a correct rendition of the correct wirings as determined by the OSGi metadata and the OSGi spec compliant resolver. The problem is that when you try to write code in D that references classes in foo, you get errors indicating forbidden access and classes/methods in foo v1.1 are not present. The problem stems from the fact that B has a foo package that is also exported, and B happens to sort before C so it appears on the classpath before D. Renaming C to be A (i.e, less than B) swaps the order and resolves this issue. Of course, this is not at all a teneable approach. I read through the previous comments (comment 16 and commment 21 in particular) and don't understand the point about JLS compliance. I do understand what you are saying but as far as I know, JLS does not define the notion of Access Rules (as we talk of the on the classpath container) so the JCKs etc would never test with them so there would never be a test that would fail. Perhaps I don't really understand the semantics of the "forbidden" rule. It currently seems to mean that the stuff is there but you are not allowed to look at it. This allows us to put up an error saying that the thing you want is there but you are not allowed to see it. This is very helpful and informative. If instead the semantics of the forbidden access rule was "hidden" then the items in question would be well and truly off the classpath (as if they weren't even there) and normal Java semantics would call for the compiler (or runtime for that matter) to continue its quest elsewhere. And, in the case here, eventually find the desired classes in C. You would lose the nice reporting but a) correctness beats reporting and b) there must be a way of still reporting helpfully when something really is not found. Regardless of the above, we really need a solution to this problem. The usecases are real. This is the way the OSGi runtime is designed and executes. PDE's value is in its ability to accurately tooling that behaviour.
Hidding types in a library is captured in bug 119419.
Note that in the scenario I described all code was in the workspace. Not a JAR in sight.
Ok, let's try to explain better. Imagine we implement a "HIDDEN" access rule, which would give you FORBIDDEN but keep looking for a better match further down on the classpath. Now, we end up with a classpath semantics no one else can match. For instance, let say you want to generate an Ant build script to compile your plugin, then you cannot express this magic rule for the Ant compiler task. FORBIDDEN/DISCOURAGED is not changing language semantics. It is only making them stricter. If some FORBIDDEN type is present on the classpath, javac would simply bind to it, and not issue a problem. We would bind to it, and issue an extra problem. Language wise though we did not change what type it did bind to. Using HIDDEN, we suddenly would allow a mode where a type is present on the buildpath, but (optionally) not considered (depending if there is a better match downstream). Thus we would suddenly change class lookup semantics in a way which prevents clients to build with another compiler than ours. This wasn't the case until now. If you were to compile these scenario with javac, then you likely would need to tune the classpath ordering until you achieve what you want. Is that unrealistic for PDE to achieve ?
To clarify. Without support HIDDEN rules, a compiler may not be able to correctly compile some code (the hidden type could cause some compile errors). This is why I am reluctant to add the HIDDEN rule. Now, we could still support this mode, and down the road, once JSR-199 is implemented, some compiler implementation could be invoked in a way which would mimic the HIDDEN rule. But until it does, we would be in some sort of proprietary mode (you can only compile Eclipse with the Eclipse compiler... is this still true Java there?).
>you likely would need to tune the classpath ordering until you achieve what >you want. Is that unrealistic for PDE to achieve ? It is realistic to achieve, but observe how it can easily break. Let's say my plugin depends on two plugins X and Y Plugin X supplies two packages: - package A: forbidden - package B: accessible Plugin Y supplies two packages: - package A: accessible - package B: forbidden. If my plug-in Z imports packages A and B, it will get wired to package A from plugin Y and package B from plugin X. In this case, no single classpath order will be good enough. if plugin X is before Y on the classpath, I get flagged for using forbidden package A from X. If plugin Y is first on the classpath, I get flagged for using forbidden package B from Y.
Ok. Talking with Jeff, we will look at implementing HIDDEN mode early M5. It will not be a rule used by default, since it ties client to Eclipse compiler. We also need to leverage this rule into batch compiler and Ant adapter. In the future, with proper compiler APIs, certain name environment implementations will be able to match this behavior. Jerome - pls look at rule addition (still need to find a good name). Then pass it onto Maxime to fix up batch and Ant. The semantic for the rule is: if a HIDDEN type is found in name environment, then remember it, but keep looking. If no subsequent type is found, then it is answered and treated as if FORBIDDEN. If another type is found, then if not HIDDEN itself, then answer this one instead (note that if there are 2 HIDDEN types in sequence, the first one always takes precedence). HIDDEN needs to mean FORBIDDEN+KEEP_SEARCHING. Need a good name for it.
Jerome and I suggest: FORBIDDEN_UNLESS_BETTER
(in the name of Philippe): SUGGEST_FORBIDDEN is a good candidate
I've got to say it. SOFT_FORBIDDEN
SUGGEST_FORBIDDEN carries the idea it is a recommendation, so that one can find a better match further down... SOFT/PROMISCUOUS_FORBIDDEN doesn't carry much information. How different is SOFT_FORBIDDEN from DISCOURAGED ? FORBIDDEN_IF_NO_BETTER ? We have search request policies which are pretty verbose, but in the end are pretty convenient: WAIT_UNTIL_INDEX_READY etc...
Hey, your component, your API, you get to pick the name :-) I'm just happy to have the function.
Ok. Jerome, pls define SUGGEST_FORBIDDEN access rule.
Phlippe, can you clarify what 'SUGGEST_FORBIDDEN' does: Is it just used for the type access restriction warnings, or does it actually influence the class lookup that is used by the compiler (comment 44)? My guess is that comment 44 is what Equinoz needs here (and also bug 119419). There's always the case where two classes have the qualified name not only have different access restrictions but also different APIs. The compiler would still complain about a method that can't be found, while things work at runtime. So isn't 'HIDDEN' is the correct term? 'SUGGEST_FORBIDDEN' sounds like this is a workaround implemented for access rule. If it is just a workaround for access rules: I would suggest to choose a better term than 'SUGGEST_FORBIDDEN'. Maybe FORBIDDEN_IF_UNIQUE?
Comment 44 is still valid. We are only scratching our heads to find a good name for it. The semantic will affect all type lookups. HIDDEN isn't a bad name, but it doesn't capture the fact it will be considered as FORBIDDEN unless there is a better match further down on the classpath. This is why SUGGEST_FORBIDDEN rather feels like a recommendation. The name isn't in stone. I don't think FORBIDDEN_IF_UNIQUE is any more clear, but I am flexible.
Also for consistency with existing APIs (IAccessRule#K_ACCESSIBLE, IAccessRule#K_NON_ACCESSIBLE), we should consider using the wording 'non accessible' or something similar, instead of 'forbidden'.
After a discussion with Philippe, the new idea is to have a flag on existing access rules (instead of addind a new kind of rule), that would caused the rule to be considered unless there is another rule on another classpath entry that match the type. What should be the name of this flag ? I propose 'keepLooking'. Any better name ?
IGNORE_IF_BETTER_MATCH_IN_NEXT_ENTRY A bit verbose, but at least no need to explain...
Added flag IAccessRule#IGNORE_IF_BETTER that indicates that the rule should be ignored if a better rule is found. E.g. if a rule K_NON_ACCESSIBLE | IGNORE_IF_BETTER matches type p.X and a rule K_DISCOURAGED that also matches p.X is found after the first one, then p.X will be reported as discouraged. Added regression tests MultiProjectTests#testIgnoreIfBetterNonAccessibleRule1/2, ReconcileTests#testIgnoreIfBetterNonAccessibleRule1-4, ClasspathTests#testEncodeDecodeEntry05 Entered bug 124090 against PDE UI so as to fix the original problem.
Verified for 3.2 M5 using build I20060214-0010.
