RE: [hyades-dev] More info on Java UTF-8

[Dave Smith <smith@xxxxxxxxxx>]

Allan,

The binding or protocol layer in the
current RAC, on EBCDIC machines, does an ASCII to EBCDIC conversion when
receiving messages and does an EBCDIC to ASCII conversion when sending
messages.  In the agent, if the string data is passed to java code
it is converted back to ASCII.  Therefore, the bytes of string data
sent by a java client should be the same as the bytes received by a java
agent.  The data is EBCDIC only while in native code.    

At the time of porting the RAC to EBCDIC
machines this seemed like a good strategy since most of the RAC native
code was built in EBCDIC encoding.  However, as you point out if the
major focus of HDCE is "java clients" connected to "java
agents", these EBCDIC/ASCII conversions are not necessary most of
the time.  And if a conversion is necessary, it can be done above
the protocol layer.   I agree with your suggestion of using UTF-8
throughout the whole new protocol.


Dave N. Smith
Data Collection Tooling              
            
IBM Toronto Laboratory
D3/RNQ/8200/MKM
External: 905-413-3916
Tieline: 969-3916
smith@xxxxxxxxxx

Allan K Pratt <apratt@xxxxxxxxxx>
Sent by: hyades-dev-admin@xxxxxxxxxxx

08/19/2004 02:49 PM

Please respond to
hyades-dev

To	hyades-dev@xxxxxxxxxxx
cc
Subject	RE: [hyades-dev] More info on Java UTF-8

Actually, I'd noticed that. Since messages like the
filter specification 
include Java class names, this implies that the filtering doesn't work
for 
non-ASCII class names on EBCDIC targets, even though such class names are
perfectly acceptable to the JVM. I would expect these class names to be
encoded as UTF-8 in the message, but the conversion to EBCDIC will fail
for non-ASCII char values. Am I right? 

This is one reason I think we should use UTF-8 throughout the new 
protocol, regardless of the local machine encoding. Transcoding should
occur only when a string enters or leaves the "protocol space."
But in the 
new protocol we're defining, a message from an Eclipse workbench to a 
Java-focused agent should never undergo transcoding to EBCDIC, even if
it's transmitted to an EBCDIC machine.

-- Allan Pratt, apratt@xxxxxxxxxx
Rational software division of IBM




Dave Smith <smith@xxxxxxxxxx> 
Sent by: hyades-dev-admin@xxxxxxxxxxx
08/19/2004 08:08 AM
Please respond to
hyades-dev


To
hyades-dev@xxxxxxxxxxx
cc

Subject
RE: [hyades-dev] More info on Java UTF-8







The only conversion the current RAC does on string data in control 
messages is EBCDIC/ASCII conversion on EBCDIC machines.  It doesn't
do any 
internationalization conversion for the local machine. 

Dave N. Smith
Data Collection Tooling 
IBM Toronto Laboratory
D3/RNQ/8200/MKM
External: 905-413-3916
Tieline: 969-3916
smith@xxxxxxxxxx




"Kaylor, Andrew" <andrew.kaylor@xxxxxxxxx> 
Sent by: hyades-dev-admin@xxxxxxxxxxx 
08/18/2004 07:35 PM 

Please respond to
hyades-dev


To
<hyades-dev@xxxxxxxxxxx> 
cc

Subject
RE: [hyades-dev] More info on Java UTF-8








I also don?t think we need to match Java?s internal representation of 
strings.  The only things we need to worry about are: 
  
1. If the data is received via a Java socket, how easily can it be 
translated into a Java string? 
2. If the data is sent via a Java socket, how easily can it be gotten from
a Java string? 
3. If the data is received via a C socket, how easily can in be translated
into a standard C format? 
4. If the data is sent via a C socket, how easily can in be gotten from
a 
standard C format? 
5. If the data is sent to an agent which is primarily implemented in Java,
how easily can it make the JNI transition between the C stub and the Java
agent? 
6. If the data is sent from an agent which is primarily implemented in
Java, how easily can it make the JNI transition between the Java agent
and 
the C stub? 
  
In the case of the third and fourth questions, a lot depends on how we
actually plan to represent character strings in the HCE code itself.  It
seems likely that we?ll be translating out of the wire format and into
some ?local? format early on in the process.  So we need to have a
strategy for having the HCE code internationalizable. 
  
Does the current RAC worry about that? 
  
-Andy 
  
-----Original Message-----
From: hyades-dev-admin@xxxxxxxxxxx [mailto:hyades-dev-admin@xxxxxxxxxxx]
On Behalf Of Nguyen, Hoang M
Sent: Wednesday, August 18, 2004 3:47 PM
To: hyades-dev@xxxxxxxxxxx
Subject: [hyades-dev] More info on Java UTF-8 
  
  
Hello all, 
  
If we want to adopt the Java UTF-8 form, we may want to consider adopting
its data structure as well. 
  
Here is the spec of UTF-8 data structure in Java Virtual Machine (JVM)
   
http://java.sun.com/docs/books/vmspec/2nd-edition/html/ClassFile.doc.html#7963

  
·         2-byte length 
·         followed by the UTF-8 byte stream 
·         length does not contain the null character
  
In addition, I have verified that Java handles the single null byte 
translation as well. 
Please see attached programs. 
  
We can discuss more and get some resolution on this issue in our weekly
meeting. 
  
Regards, 
  
  
  
/* 
   This demo program shows that Java can handle 
      UTF-8 file with null byte is translated as one byte.
 */ 
  
import java.io.* ; 
  
public class MyUTF8Output 
{ 
     public static void main(String args[]) 
     { 
          FileOutputStream    fos ;
  
          OutputStreamWriter  osw ; 
  
          char[] msg = {'A', '\u0000', 'B', '\u0080',
'C', '\u0000'} ; 
  
          try 
          { 
              String s = new String(msg)
; 
  
              fos = new FileOutputStream("myoutput.txt");
  
              osw = new OutputStreamWriter(fos,
"UTF-8"); 
  
              osw.write(s) ; 
 
              osw.flush() ; 
  
              fos.close(); 
        
              System.out.println("See
\"myoutput.txt\" file.") ; 
          } 
          catch (Exception e) { }    
  
    } 
} 
  
  
  
/* 
   This demo program shows that Java UTF-8 format is: 
        - 2-byte   leng of the UTF-8 buffer 
        - null byte is mapped into two bytes 
 */ 
  
import java.io.* ; 
  
public class MyUTF8Conversion 
{ 
     public static void main(String args[]) 
     { 
          FileOutputStream    fos ;
  
          DataOutputStream    dos ;
  
          char[] msg = {'A', '\u0000', 'B', '\u0080',
'C', '\u0000'} ; 
  
          try 
          { 
              String s = new String(msg)
; 
  
              fos = new FileOutputStream("myoutput2.txt");
  
              dos = new DataOutputStream(fos);
  
              dos.writeUTF(s) ; 
 
              dos.flush() ; 
  
              fos.close(); 
        
              System.out.println("See
\"myoutput2.txt\" file.") ; 
          } 
          catch (Exception e) { }    
  
    } 
} 
  
  

_______________________________________________
hyades-dev mailing list
hyades-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/hyades-dev