Comment 1 Ivar Meikas 2011-03-15 13:42:00 EDT

Created attachment 191230 [details]
Temporary fix for exceptions

Comment 2 Reiner Hille 2011-10-14 05:59:23 EDT

I'm irritated by the issue and will spend some time looking for it. Normally EMF should simply create a proxy EObject that will be resolved in the moment when the reference is touched.

Comment 3 Benedikt Ritter 2012-03-24 13:28:50 EDT

I have spend a hole day to investigate the issue, debugging back and forth. Then I decided to take a step back and discuss with you, what I have found out so far. 

The problem in this case is the structureRef of an ItemDefinition. In the case of the Incident Management Process, we have three such ItemDefinitions (resulting in three errors):
 * IssueReport
 * TroubleTicket
 * Answer
all the StrutureRefs point to java classes (java is declared as type langague in the document root)

Here is what happens during:
XMLHandler.setValueFromId(EObject object, EReference eReference, String ids) tries to handle the structureRef attribute of and ItemDefinition. It will not create a proxy, because ids is no valid URI. Instead a SingelReference will be created and added to the forwardSingleReferences.
Then some time later on we XMLHandler.handleForwardReferences(boolean isEndDocument) gets called. When the SingleReferences get processed, no EObject can be retrieved from the xmlSource, and that causes the errors.

Next I looked into the BPMN Spec. Here are the relevant parts of BPMN 2.0 Spec, 8.3.10 Item Definition:

"The structure attribute references the actual data structure."

"An Import is used to further identify the location of the data structure (if applicable). For example, in the case of data structures contributed by an XML schema, an Import would be used to specify the file location of that schema."

"An ItemDefinition element can specify an import reference where the proper definition of the structure is defined."

If you have a look at Figure 8.25 - ItemDefinition class diagram you can see, that structureRef is of type Element (not of type BaseElement or RootElement!) I think this is the important point. It is not clearly specified what a structureRef is. 

The ecore model declares StructureRefs as a collection of EObjects. In other words: StructureRefs are entities that reside in an ecore model (according to the bpmn2 ecore model). But this does not have to be the case. StrutureRefs can be anything.
In the case of the Incident Management Process, we want to use Java classes as data structure. There is no import, because a java class is identified by it's fully qualified name (at least in one classpath (if you are not using OSGi...)). So I guess, we have to decide how to handle this.

Comment 4 Benedikt Ritter 2012-03-25 05:55:07 EDT

I think the easiest way to fix this bug is to simply declare structureRef as an EString. 

There is one constraint for ItemDefinitions mentioned on p. 92 of BPMN Spec 2.0: "In cases where the data structure represents a collection, the multiplicity can be projected into the attribute isCollection. If this attribute is set to “true,” but the actual type is not a collection type, the model is considered as invalid. BPMN compliant tools might support an automatic check for these inconsistencies and report this as an error."

I really doubt, that this can be easily implemented. Take for example the incident process. There we have an ItemDefinition like the following:
<itemDefinition id="IssueItem" isCollection="false" itemKind="Information"
   structureRef="com.camunda.examples.incidentmanagement.IssueReport" />
A tool that has to validate whether or not the constraint is violated, would have to recognize com.camunda.examples.incidentmanagement.IssueReport as the identifier of a java class. Then that class has to be found (where?) and it has to be inspected whether or not it is a collection (how? does it have to implement Iterable<T>? or Collection<T>?).
If the structureRef refers to an XML Schema there has to be a hole different validation.

Comment 5 Reiner Hille 2012-03-26 06:28:13 EDT

I have quickly checked the problem. First the short answer: The file is bad, because the complained structureRef entries are no QName. They are Java Qualified names, but not XSD QNames. If I modify file and put a ":" in front of the Java qualified name, it works:
 <interface name="Product Backlog Interface"          implementationRef="java:com.camunda.examples.incidentmanagement.ProductBacklog">


Here the long answer:
1. structureRef is a cross reference. In BPMN 2.0 CMOF it points to "ModelElement", which is the most generic untyped reference.
2. In the schema there is QName, meaning there is a prefixed name with a declared XML namespace. 
3. In ECore I have always a reference to EObject with resolveProxies = true. Why? Because I believe the a cross reference should be managed by EMF. This e.g. works great if your structure defintion is in a format that EMF also understand, e.g. as an XSD or WSDL file. 
4. Unfortunately in EMF if the reference is EObject, also the Java type is EObject. I understand that Java programmers would like to have here anything.
5. Using EString as type is not a good idea from my oppion. This means that the former reference not becomes an attribute and the in any case the end user needs to care for references. In above cases this is really a big disadvantage.
6. It is perfectly fine to have something in the "local" part of QName, that EMF does not understand. In such case EMF would simply create a proxy and the user could resolve it as he wants. E.g. in above case (pseudocode):

if (interface.getImplementationRef().eIsProxy) {
 String qName = interface.getImplementationRef().eProxyURI(); 
 String[] parts = qName.split(":");
 String javaName = parts[1];
}

7. There seems to be a problem in EMF that is tries to condider an attribute as idref (not QName) if there is no colon in the name, which caused the problem here. This happens, even if BPMN knows that the field MUST be a QName. I'm currently not sure if the problem comes from EMF itself or from our QNameURIConverter. In second case me might have the chance to fix the problem and make our reader more tolerant in "incorrect" QNames.

7. Even if in this case the developer would get the original entry from the proxy's URI, it would be cool if there would be an elegant way to resolve it in the model. We would need:
7a.: An extension point that allows to hook into proxy resolving for those kind of "pseudo" URI that in fact contain private strings. A user would hook in hins implementatation that resolves that object from his string.
7b: A place where the resolved object is stored. Again: unfortunately the reference is EObject. If it was Object, it could take aything. We could either have an addition field in ItemDefinition, e.g. javaStructureRef with type EJavaObject. Or we use one of the new genModel flags that redirect ECore types to Java types. I have recently seen such cases where EMF replaced EObject with Object.


Reiner.

Comment 6 Benedikt Ritter 2012-03-26 07:22:35 EDT

Hi Reiner,

thanks for your long answer. It always takes some time to get to now a code base. Your help is really appreciated!

(In reply to comment #5)
> I have quickly checked the problem. First the short answer: The file is bad,
> because the complained structureRef entries are no QName. They are Java
> Qualified names, but not XSD QNames. If I modify file and put a ":" in front of
> the Java qualified name, it works:
>  <interface name="Product Backlog Interface"         
> implementationRef="java:com.camunda.examples.incidentmanagement.ProductBacklog">
> 

We should have a look (and maybe file a bug) for the properties view. It does not show anything for QName Refs outside the model. It is not possible to type in any values that don't exist as EObjects inside the model. Also, the list of possible values does contain all the stuff, that you usually would not want to use as a structurRef (Definitions, contents from DI...)

> 
> Here the long answer:
> 1. structureRef is a cross reference. In BPMN 2.0 CMOF it points to
> "ModelElement", which is the most generic untyped reference.
> 2. In the schema there is QName, meaning there is a prefixed name with a
> declared XML namespace. 
> 3. In ECore I have always a reference to EObject with resolveProxies = true.
> Why? Because I believe the a cross reference should be managed by EMF. This
> e.g. works great if your structure defintion is in a format that EMF also
> understand, e.g. as an XSD or WSDL file. 
> 4. Unfortunately in EMF if the reference is EObject, also the Java type is
> EObject. I understand that Java programmers would like to have here anything.
> 5. Using EString as type is not a good idea from my oppion. This means that the
> former reference not becomes an attribute and the in any case the end user
> needs to care for references. In above cases this is really a big disadvantage.

I see no way to reconstruct the Incident Management Model, if structureRef is an EObject. Maybe I have not understand the problem completely yet. But how can you declare a structureRef to java:com.camunda.examples.incidentmanagement.IssueReport with the current implementation of the editor?

> 6. It is perfectly fine to have something in the "local" part of QName, that
> EMF does not understand. In such case EMF would simply create a proxy and the
> user could resolve it as he wants. E.g. in above case (pseudocode):
> 
> if (interface.getImplementationRef().eIsProxy) {
>  String qName = interface.getImplementationRef().eProxyURI(); 
>  String[] parts = qName.split(":");
>  String javaName = parts[1];
> }
> 
> 7. There seems to be a problem in EMF that is tries to condider an attribute as
> idref (not QName) if there is no colon in the name, which caused the problem
> here. This happens, even if BPMN knows that the field MUST be a QName. I'm
> currently not sure if the problem comes from EMF itself or from our
> QNameURIConverter. In second case me might have the chance to fix the problem
> and make our reader more tolerant in "incorrect" QNames.

I looked at that class for a while, during debugging yesterday, but nothing caught my eye. I'll have a look at it again. I probably missed something. 

> 
> 7. Even if in this case the developer would get the original entry from the
> proxy's URI, it would be cool if there would be an elegant way to resolve it in
> the model. We would need:
> 7a.: An extension point that allows to hook into proxy resolving for those kind
> of "pseudo" URI that in fact contain private strings. A user would hook in hins
> implementatation that resolves that object from his string.

Sounds awesome! Let's do it :-)

> 7b: A place where the resolved object is stored. Again: unfortunately the
> reference is EObject. If it was Object, it could take aything. We could either
> have an addition field in ItemDefinition, e.g. javaStructureRef with type
> EJavaObject. Or we use one of the new genModel flags that redirect ECore types
> to Java types. I have recently seen such cases where EMF replaced EObject with
> Object.
> 

I tried that some while ago and it did not work out to well. But I only gave it 2 hours or something. We should invest some more time and see if it works for us.

> 
> Reiner.

looking forward to your reply!
Benedikt

Comment 7 Benedikt Ritter 2012-03-26 17:07:39 EDT

Hey again,

I'm getting closer to the root cause. The problem is, that neither com.camunda.examples.incidentmanagement.IssueReport nor java:com.camunda.examples.incidentmanagement.IssueReport will be converted to URI by QNameURIHandler because of lines 65-69:

if (fragment.contains(".")) {
       // HACK: officially IDs can contain ".", but unfortunately XmlHandler calls resolve also for xsi:schemaLocation stuff and similar, that are
       // NO URIs. We must not process them.
       return qName;
}

Now, depending on the presence of a prefix, XMLHandler.setValueFromId(EObject object, EReference eReference, String ids) will handle the element differently. If a prefix is present, line 2775: 

else if (id.indexOf(':', 0) != -1)

will evaluate to true, skipping all the SingleReference stuff that follows. If no prefix is present, that condition evaluates to false and a SingleReference will be created, that later on can not be resolved by XMLHandler.handleForwardReferences(boolean isEndDocument), causing the exceptions.

Thoughts?

Comment 8 Henning Heitkoetter 2012-03-28 09:28:52 EDT

Do I understand correctly that we could solve the problem if we find a better check to replace fragment.contains(".")?

In that case, we should check for a slash "/" instead. A slash cannot be part of a QName, in contrast to a dot. Hence, if the URI contains a slash, it is not used to represent a QName.* However, I'm not sure if this alternative check covered all use cases that Reiner had in mind with the check. At least xsi:schemaLocation would be covered.

* Background: we use URIs to represent QNames, because every QName is a syntactically valid URI (NOT vice versa), with the prefix of a QName matching the scheme part of an absolute URI. An unprefixed QName corresponds to a relative URI.

Comment 9 Benedikt Ritter 2012-03-29 14:05:14 EDT

(In reply to comment #8)
> Do I understand correctly that we could solve the problem if we find a better
> check to replace fragment.contains(".")?
> 

As far as I understand, yes that is correct. For example using com-camunda-examples-incidentmanagement-IssueReport as structureRef works.

> In that case, we should check for a slash "/" instead. A slash cannot be part
> of a QName, in contrast to a dot. Hence, if the URI contains a slash, it is not
> used to represent a QName.* However, I'm not sure if this alternative check
> covered all use cases that Reiner had in mind with the check. At least
> xsi:schemaLocation would be covered.
> 

That maybe a solution. But let's hear what Reiner has to say to this.

> * Background: we use URIs to represent QNames, because every QName is a
> syntactically valid URI (NOT vice versa), with the prefix of a QName matching
> the scheme part of an absolute URI. An unprefixed QName corresponds to a
> relative URI.

Comment 10 Reiner Hille 2012-03-29 17:04:22 EDT

Hi,
as I'm currently offsite, could one of you check if it's my code that makes the java class name considered as an idref or if this is in EMF itself? I mean does my dot-search hack make the things worse here? What happens if someone uses a simple string without dot and without colon as URI here? Wouldn't it be also considered an idref?
About Hennings proposal: Unfortunately I don't rember anymore the details about the hack, but I had a sample that caused problems with schema location stuff. Maybe this is still available in one of the tests or closed bugs. Maybe slash would also work.

Reiner.

Comment 11 Henning Heitkoetter 2012-03-30 10:40:29 EDT

Indeed, there is a test for this: testExternalTypedExtension in ExtensibilityTests. The corresponding file TypedExtension.bpmn2 has a xsi:schemaLocation attribute with value "http://www.acme.com/bpmn-ext Example-Extension.xsd". This specific test succeeds with the dot-check in place and fails otherwise. It also fails when checking for a slash*, because the second URI does not contain a slash. In general, however, neither of these checks will work, because schemaLocation may contain arbitrary URIs (with or without dot, although the latter might be quite rare), even those that could be a valid QName (like Example-Extension.xsd). A check for a single character will never be sufficient to catch all cases.

We should find a way to avoid converting the URIs of this field and similar ones, where we don't expect to find a QName. Up to now, I haven't found an easy one :(

Note: * Which we already do in the first line, BTW :)