Description Tomer Mahlin 2008-08-25 11:31:11 EDT

Background
===========

"... Bidirectional text can be stored in several formats (a.k.a. "layouts"), depending on the platform and the intended use. When Bidi data is available but not in the appropriate layout, it is possible to convert it to the required layout by performing a "Bidi transformation".  

Bidi text can be stored in a variety of layouts characterized by a combination of 5 attributes:
  * ordering scheme
  * base direction
  * symmetric swapping
  * text shaping (for Arabic)
  * numeric shaping (for Arabic)

Historically, data stored on mainframes (zSeries and iSeries) was in "visual" layout, i.e. according to a visual ordering scheme, ready for presentation on "dumb" devices like terminals or printers. With the advent of processing power closer to end users, as is the case for workstations and personal computers, it became common to store data in a newer "logical" layout ..." - Matitiahu Allouche, Bidi Architect

"... Currently, data in visual layout are still preponderant on zSeries and iSeries, while in Windows systems most data are created and processed in logical layout. Systems running Unix or its variants can handle data in either logical or visual layout, probably with some more affinity for the logical layout.
Java GUI components expect Bidi textual data to be in logical layout ..." - Matitiahu Allouche, Bidi Architect

The essence of the problem
===========================

"...What happens if some data need processing via Java / Eclipse GUI, but are currently accessible in a visual layout? They must be converted to a logical layout ..." - Matitiahu Allouche, Bidi Architect. If this transformation is not performed the data will be corrupted. This is exactly what happens in DTP.

What does Bidi transformation do ?
==================================

"... In general, the operation which converts a piece of Bidi text from a source layout to a target layout is called a "Bidi transformation".  This can include going from a logical layout to a visual layout, or going the opposite way. The transformation may even leave the ordering scheme unchanged, but alter some or all of the other Bidi attributes ..." - Matitiahu Allouche, Bidi Architect.
 Bidi transformation of text from one Bidi layout (aka format) to another results in changes of directionality (relative position of characters or segments of text) and possibly in changes in shaping. Bidi transformation is applicable not only to BiDi or mixed scripts but to exclusively Latin ones as well. When transforming text from Logical left-to-right to Visual right-to-left Latin text is mirrored. For example: "hello world" in Logical left-to-right becomes "dlrow olleh" in Visual right-to-left.


Problems with current level of support for Bidi transformation in DTP
=====================================================================

Currently there are two major problems with support for Bidi transformation in DTP:

1. DTP does not have any built-in support for Bidi transformation. Instead
it relies mainly on the Bidi support in JDBC driver. Please notice, that only minor fraction all available JDBC drivers has support for Bidi transformation 
(and even in those it is very limited). Unfortunately DTP is not capable of leveraging even this limited support for Bidi transformation which exist in some of JDBC drivers. This is mainly due to the fact, that DTP is not aware of level of support for Bidi transformation on JDBC level and thus it fails to correctly leverage this support.

2. Even those JDBC drivers which provide support for Bidi transformation (i.e. AS400JDBCDriver as part of JTOpen - the open source version of IBM Toolbox for Java available from http://jt400.sourceforge.net/) have only very limited version of this support. Consequently, since DTP relies on JDBC driver with respect to support for Bidi transformation, it does not provide flexible enough support for Bidi transformation.


What other products do ?
========================

 In order to clarify that DTP is not expected to set any precedent I would like to give an illustrative example of a product using JDBC layer for support for connectivity and data exchange with RDBMS systems. This product does not rely on support for Bidi transformation in JDBC driver (due to the fact that in many cases JDBC drivers don't have such support or have only very limited one). Instead it provides this support on the product level. For example during configuration of JDBC driver end user can provide additional parameters specifying Bidi layout properties of RDBMS data. When a connection to RDBMS is established those properties are used by this product to correctly transform Bidi data from one Bidi layout to another. For illustration please see attached snapshot (How other products providing similar functionality address support for Bidi transformation). 

Was Bidi transformation itself already implemented ?
====================================================

 Performing Bidi transformation "... is not trivial.  It entails reversing parts of the text, but not all of it.  It entails replacing some occurrences of symmetric symbols by their symmetric match. For Arabic, it also entails choosing the proper shape for each letter and the proper digits for each number. Fortunately, software tools already exist to accomplish such conversions ..." - Matitiahu Allouche, Bidi Architect.
  "...Software tools which perform Bidi transformations are called "Bidi engines".  Several such Bidi engines are available to Eclipse developers.  They all accomplish more or less the same tasks, the differences are mostly in the APIs.  However, there is some variance in the options offered by each engine. The main Bidi engines are:
   * the Bidi support in ICU4J
   * the Bidi engine in IBM's JDK
   * the Bidi class in Sun's JDK
..." - Matitiahu Allouche, Bidi Architect.

What is missing in DTP ?
========================

 DTP is expected not to rely on the support for Bidi transformation in JDBC drivers (mainly due to the fact that in overwhelming majority of cases JDBC drivers don't have such support or have only very limited one). Instead it is expected to provide built-in support. 
 However, DTP does not need to implement Bidi transformation itself (since it can leverage implementations available from ICU4J/JDK). What it needs to do is comprised of following major tasks:

1. Provide configuration option to the end user in order to define Bidi layout information associated with data. For example, Bidi layout associated with connection, database, meta data (such as table name, column name etc.) and content data (actual data stored in the database).

2. Extend appropriate model to store this configuration data.

3. During data exchange with RDBMS use this configuration data and provide Bidi transformation of data from one Bidi layout to another. During data exchange with RDBMS system DTP can leverage any one of existing tools (ICU4J, JDK) to perform actual Bidi transformation of data.