Attachment #200742 for bug #183164

View | Details | Raw Unified | Return to bug 183164 | Differences between

and this patch




/*******************************************************************************
 * Copyright (c) 2010, 2011 IBM Corporation and others.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *     IBM Corporation - initial API and implementation
 ******************************************************************************/
package org.eclipse.equinox.bidi;

import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.internal.STextImpl;

/**
 * For a general introduction to structured text, see
 * {@link <a href="package-summary.html"> the package documentation</a>}.
 * <p>
 * Several common processors are included in <b>STextEngine</b>. For processors 
 * supplied by other packages, a processor instance can be obtained using the
 * {@link org.eclipse.equinox.bidi.STextProcessorFactory#getProcessor}
 * method for the registered processors, or by instantiating a private processor.
 * </p><p>
 * Most of the methods in this class have a <code>text</code>
 * argument which may be just a part of a larger body of text.
 * When it is the case that the text is submitted in parts with
 * repeated calls, there may be a need to pass information from
 * one invocation to the next one. For instance, one invocation
 * may detect that a comment or a literal has been started but
 * has not been completed. In such cases, a <code>state</code>
 * argument must be used.
 * </p><p>
 * The <code>state</code> argument must be an array of integers
 * with at least one element. Only the first element is used by
 * the methods of this class.
 * </p><p>
 * When submitting the initial part of the text, the first element
 * of <code>state</code> must contain the value {@link #STATE_INITIAL}
 * or any value <= 0.
 * </p><p>
 * After calling a method with a non-null <code>state</code> argument,
 * a value is returned in the first element of <code>state</code>. This
 * value should be passed unmodified to the method when calling it again
 * with the text which is the continuation of the text submitted in the
 * last call.
 * </p><p>
 * When the text submitted to a method is not a continuation and is not
 * expected to have a continuation , e.g. it is processed all by itself,
 * the <code>state</code> argument should be specified as <code>null</code>.
 * </p><p>
 * <b>Code Samples</b>
 * </p><p>
 * The following code shows how to transform a certain type of structured text
 * (directory and file paths) in order to obtain the <i>full</i>
 * text corresponding to the given <i>lean</i> text.
 * <pre>
 *   String leanText = "D:\\\u05d0\u05d1\\\u05d2\\\u05d3.ext";
 *   String fullText = STextEngine.leanToFullText(STextEngine.PROC_FILE, null, leanText, null);
 *   System.out.println("full text = " + fullText);
 * </pre>
 * </p><p>
 * The following code shows how to transform successive lines of Java
 * code in order to obtain the <i>full</i>
 * text corresponding to the <i>lean</i> text of each line.
 * <pre>
 *   int[] state = new int[1];
 *   state[0] = STextEngine.STATE_INITIAL;
 *   String leanText = "int i = 3; // first Java statement";
 *   String fullText = STextEngine.leanToFullText(STextEngine.PROC_JAVA, null, leanText, state);
 *   System.out.println("full text = " + fullText);
 *   leanText = "i += 4; // next Java statement";
 *   fullText = STextEngine.leanToFullText(STextEngine.PROC_JAVA, null, leanText, state);
 *   System.out.println("full text = " + fullText);
 * </pre>
 * </p>
 *  @author Matitiahu Allouche
 *
 */
public class STextEngine {

	/**
	 *  Constant specifying that the base direction of a structured text is LTR.
	 *  The base direction may depend on whether the GUI is
	 *  {@link STextEnvironment#getMirrored mirrored} and may
	 *  may be different for Arabic and for Hebrew.
	 *  This constant can appear as value returned by the
	 *  {@link #getCurDirection getCurDirection} method.
	 */
	public static final int DIR_LTR = 0;

	/**
	 *  Constant specifying that the base direction of a structured text is RTL.
	 *  The base direction may depend on whether the GUI is
	 *  {@link STextEnvironment#getMirrored mirrored} and may
	 *  may be different for Arabic and for Hebrew.
	 *  This constant can appear as value returned by the
	 *  {@link #getCurDirection getCurDirection} method.
	 */
	public static final int DIR_RTL = 1;

	/**
	 *  Constant to use in the first element of the <code>state</code>
	 *  argument when calling most methods of this class
	 *  to indicate that there is no context of previous lines which
	 *  should be initialized before performing the operation.
	 */
	public static final int STATE_INITIAL = 0;

	private static final int[] EMPTY_INT_ARRAY = new int[0];

	/**
	 *  Prevent creation of a STextEngine instance
	 */
	private STextEngine() {
		// nothing to do
	}

	/** 
	 * Add directional formatting characters to a structured text
	 * to ensure correct presentation.
	 * 
	 * @param  processor the processor applicable to the text. If <code>null</code>, 
	 * the method returns unmodified text.
	 * 
	 * @param  environment a bidi environment. If <code>null</code>, the default environment 
	 * is used.
	 *  
	 * @param text is the structured text string
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return the structured text with directional formatting characters added to ensure 
	 * correct presentation.
	 */
	public static String leanToFullText(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null)
			return text;
		return STextImpl.leanToFullText(processor, environment, text, state);
	}

	/**
	 * Given a <i>lean</i> string, compute the positions of each of its
	 * characters within the corresponding <i>full</i> string.
	 *
	 * @param  processor designates a processor instance. If <code>null</code>, this 
	 * method returns an identity map.
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string.
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return an array which specifies offsets of the <code>text</code> characters
	 * in the <i>full</i> string
	 */
	public static int[] leanToFullMap(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null) {
			int[] map = new int[text.length()];
			for (int i = 0; i < map.length; i++)
				map[i] = i;
			return map;
		}
		return STextImpl.leanToFullMap(processor, environment, text, state);
	}

	/**
	 * Given a <i>lean</i> string, compute the offsets of characters
	 * before which directional formatting characters must be added
	 * in order to ensure correct presentation.
	 * <p>
	 * Only LRMs (for a string with LTR base direction) and RLMs (for
	 * a string with RTL base direction) are considered. Leading and
	 * trailing LRE, RLE and PDF which might be prefixed or suffixed
	 * depending on the {@link STextEnvironment#getOrientation orientation} of the
	 * GUI component used for display are not reflected in this method.
	 * </p>
	 * @param processor designates a processor instance
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return an array of offsets to the characters in the <code>text</code> argument 
	 * before which directional marks must be added to ensure correct presentation.
	 * The offsets are sorted in ascending order.
	 */
	public static int[] leanBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null)
			return EMPTY_INT_ARRAY;
		return STextImpl.leanBidiCharOffsets(processor, environment, text, state);
	}

	/**
	 * Remove directional formatting characters which were added to a
	 * structured text string to ensure correct presentation.
	 *
	 * @param  processor designates a processor instance
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string including directional formatting characters.
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return the structured text string without directional formatting characters 
	 * which might have been added by processing it with {@link #leanToFullText}.
	 *
	 */
	public static String fullToLeanText(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null)
			return text;
		return STextImpl.fullToLeanText(processor, environment, text, state);
	}

	/**
	 * Given a <i>full</i> string, compute the positions of each of its
	 * characters within the corresponding <i>lean</i> string.
	 *
	 * @param  processor designates a processor instance
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string including directional formatting characters.
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return an array of integers with one element for each of the characters
	 * in the <code>text</code> argument, equal to the offset of the corresponding character 
	 * in the <i>lean</i> string. If there is no corresponding character in the <i>lean</i> string 
	 * (because the specified character is a directional formatting character added when invoking 
	 * {@link #leanToFullText}), the value returned for this character is -1.
	 */
	public static int[] fullToLeanMap(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null) {
			int[] map = new int[text.length()];
			for (int i = 0; i < map.length; i++)
				map[i] = i;
			return map;
		}
		return STextImpl.fullToLeanMap(processor, environment, text, state);
	}

	/**
	 * Given a <i>full</i> string, return the offsets of characters
	 * which are directional formatting characters that have been added
	 * in order to ensure correct presentation.
	 * <p>
	 * LRMs (for a string with LTR base direction), RLMs (for a string with RTL base direction) 
	 * are considered as well as leading and trailing LRE, RLE and PDF which might be prefixed 
	 * or suffixed depending on the {@link STextEnvironment#getOrientation orientation} 
	 * of the GUI component used for display.
	 * </p>
	 * @param  processor designates a processor instance
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string including directional formatting characters
	 *
	 * @param  state can be used to specify that the <code>text</code> argument is 
	 * the continuation of text submitted in a previous call and/or to receive information 
	 * to pass to continuation calls. If all calls to this method are independent from one another,
	 * this argument should be specified as <code>null</code>.
	 *
	 * @return an array of offsets to the characters in the <code>text</code> argument which 
	 * are directional formatting characters added to ensure correct presentation. The offsets 
	 * are sorted in ascending order.
	 */
	public static int[] fullBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
		if (processor == null)
			return EMPTY_INT_ARRAY;
		return STextImpl.fullBidiCharOffsets(processor, environment, text, state);
	}

	/**
	 * Get the base direction of a structured text. This base direction may depend on
	 * whether the text contains Arabic or Hebrew words. If the text contains both, 
	 * the first Arabic or Hebrew letter in the text determines which is the governing script.
	 *
	 * @param  processor designates a processor instance
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string
	 *
	 * @return the base direction of the structured text, {@link #DIR_LTR} or {@link #DIR_RTL}
	 */
	public static int getCurDirection(STextProcessor processor, STextEnvironment environment, String text) {
		if (processor == null)
			return DIR_LTR;
		return processor.getDirection(environment, text);
	}

}

Lines 13-20 Link Here

(-)src/org/eclipse/equinox/bidi/STextEnvironment.java (-2 / +2 lines)
13	import org.eclipse.equinox.bidi.internal.STextActivator;	13	import org.eclipse.equinox.bidi.internal.STextActivator;
14		14
15	/**	15	/**
16	* This class describes environment within which structured text strings are	16	* This class describes the environment within which structured text strings
17	* processed. It includes, for example:	17	* are processed. It includes:
18	* <ul>	18	* <ul>
19	* <li>locale,</li>	19	* <li>locale,</li>
20	* <li>desired orientation,</li>	20	* <li>desired orientation,</li>




/*******************************************************************************
 * Copyright (c) 2010, 2011 IBM Corporation and others.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *     IBM Corporation - initial API and implementation
 ******************************************************************************/
package org.eclipse.equinox.bidi;

import org.eclipse.equinox.bidi.custom.*;
import org.eclipse.equinox.bidi.internal.STextImpl;

/**
 * For a general introduction to structured text, see
 * {@link <a href="package-summary.html"> the package documentation</a>}.
 * <p>
 * Several common processors are included in 
 * {@link org.eclipse.equinox.bidi.STextProcessorFactory}.
 * For processors 
 * supplied by other packages, a processor instance can be obtained using the
 * {@link org.eclipse.equinox.bidi.STextProcessorFactory#getProcessor}
 * method for the registered processors, or by instantiating a private processor.
 * </p><p>
 * Most of the methods in this class have a <code>text</code>
 * argument which may be just a part of a larger body of text.
 * When it is the case that the text is submitted in parts with
 * repeated calls, there may be a need to pass information from
 * one invocation to the next one. For instance, one invocation
 * may detect that a comment or a literal has been started but
 * has not been completed.
 * </p><p>
 * This information is summarized as one integer, the "<b>state</b>".
 * The <b>state</b> can be queried using {@link #getState} after any
 * operation, and it can be set before an operation using 
 * {@link #setState}.
 * </p><p>
 * When submitting the initial part of the text, the <b>state</b>
 * must contain the value {@link #STATE_INITIAL} or any value <= 0.
 * </p><p>
 * When calling a method repeatedly, the <b>state</b> is automatically 
 * passed from one invocation to the next unless it is reset
 * by the caller using {@link #setState}.
 * </p><p>
 * <b>Code Samples</b>
 * </p><p>
 * The following code shows how to transform a certain type of structured text
 * (directory and file paths) in order to obtain the <i>full</i>
 * text corresponding to the given <i>lean</i> text.
 * <pre>
 *   STextHandler handler = new STextHandler(STextProcessorFactory.PROC_FILE);
 *   String leanText = "D:\\\u05d0\u05d1\\\u05d2\\\u05d3.ext";
 *   String fullText = handler.leanToFullText(null, leanText);
 *   System.out.println("full text = " + fullText);
 * </pre>
 * </p><p>
 * The following code shows how to transform successive lines of Java
 * code in order to obtain the <i>full</i>
 * text corresponding to the <i>lean</i> text of each line.
 * <pre>
 *   STextHandler handler = new STextHandler(STextProcessorFactory.PROC_JAVA);
*   String leanText = "int i = 3; // first Java statement";
 *   String fullText = handler.leanToFullText(null, leanText);
 *   System.out.println("full text = " + fullText);
 *   leanText = "i += 4; // next Java statement";
 *   fullText = handler.leanToFullText(null, leanText);
 *   System.out.println("full text = " + fullText);
 * </pre>
 * </p>
 *  @author Matitiahu Allouche
 *
 */
public class STextHandler {

	/**
	 *  Constant specifying that the base direction of a structured text is LTR.
	 *  The base direction may depend on whether the GUI is
	 *  {@link STextEnvironment#getMirrored mirrored} and may
	 *  may be different for Arabic and for Hebrew.
	 *  This constant can appear as value returned by the
	 *  {@link #getCurDirection getCurDirection} method.
	 */
	public static final int DIR_LTR = 0;

	/**
	 *  Constant specifying that the base direction of a structured text is RTL.
	 *  The base direction may depend on whether the GUI is
	 *  {@link STextEnvironment#getMirrored mirrored} and may
	 *  may be different for Arabic and for Hebrew.
	 *  This constant can appear as value returned by the
	 *  {@link #getCurDirection getCurDirection} method.
	 */
	public static final int DIR_RTL = 1;

	/**
	 *  Constant to use in {@link #setState}
	 *  to indicate that there is no context of previous lines
	 *  to consider before performing the next operation.
	 */
	public static final int STATE_INITIAL = 0;

	private static final int[] EMPTY_INT_ARRAY = new int[0];

	private STextProcessorData procData;
	private int state;

	/**
	 *  Constructor
	 *  
	 *  @param processor is the processor which will transform the 
	 *  submitted text.
	 */
	public STextHandler(STextProcessor processor) {
		procData = new STextProcessorData();
		setProcessor(processor);
		procData.handler = this;
		procData.offsets = new STextOffsets();
	}

	/**
	 *  Replace the processor of an existing handler. This operation
	 *  resets the state to {@link #STATE_INITIAL}.
	 *  
	 *  @param processor is the new processor.
	 */
	public void setProcessor(STextProcessor processor) {
		procData.processor = processor;
		state = STATE_INITIAL;
	}

	/**
	 *  Set the state for the next operation.
	 *  
	 *  @param state is the new value.
	 */
	public void setState(int state) {
		this.state = state;
	}

	/**
	 *  Get the value of the state after the last operation.
	 *  
	 *  @return the last value of the state.
	 */
	public int getState() {
		return state;
	}

	/** 
	 * Add directional formatting characters to a structured text
	 * to ensure correct presentation.
	 * 
	 * @param  environment a bidi environment. If <code>null</code>, the default environment 
	 * is used.
	 *  
	 * @param text is the structured text string
	 *
	 * @return the structured text with directional formatting characters added to ensure 
	 * correct presentation.
	 */
	public String leanToFullText(STextEnvironment environment, String text) {
		if (procData.processor == null)
			return text;
		procData.environment = environment;
		procData.text = text;
		return STextImpl.leanToFullText(procData);
	}

	/**
	 * Given a <i>lean</i> string, compute the positions of each of its
	 * characters within the corresponding <i>full</i> string.
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string.
	 *
	 * @return an array which specifies offsets of the <code>text</code> characters
	 * in the <i>full</i> string
	 */
	public int[] leanToFullMap(STextEnvironment environment, String text) {
		if (procData.processor == null) {
			int[] map = new int[text.length()];
			for (int i = 0; i < map.length; i++)
				map[i] = i;
			return map;
		}
		procData.environment = environment;
		procData.text = text;
		return STextImpl.leanToFullMap(procData);
	}

	/**
	 * Given a <i>lean</i> string, compute the offsets of characters
	 * before which directional formatting characters must be added
	 * in order to ensure correct presentation.
	 * <p>
	 * Only LRMs (for a string with LTR base direction) and RLMs (for
	 * a string with RTL base direction) are considered. Leading and
	 * trailing LRE, RLE and PDF which might be prefixed or suffixed
	 * depending on the {@link STextEnvironment#getOrientation orientation} of the
	 * GUI component used for display are not reflected in this method.
	 * </p>
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string
	 *
	 * @return an array of offsets to the characters in the <code>text</code> argument 
	 * before which directional marks must be added to ensure correct presentation.
	 * The offsets are sorted in ascending order.
	 */
	public int[] leanBidiCharOffsets(STextEnvironment environment, String text) {
		if (procData.processor == null)
			return EMPTY_INT_ARRAY;
		procData.environment = environment;
		procData.text = text;
		return STextImpl.leanBidiCharOffsets(procData);
	}

	/**
	 * Remove directional formatting characters which were added to a
	 * structured text string to ensure correct presentation.
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param text is the structured text string including directional formatting characters.
	 *
	 * @return the structured text string without directional formatting characters 
	 * which might have been added by processing it with {@link #leanToFullText}.
	 *
	 */
	public String fullToLeanText(STextEnvironment environment, String text) {
		if (procData.processor == null)
			return text;
		procData.environment = environment;
		procData.text = text;
		return STextImpl.fullToLeanText(procData);
	}

	/**
	 * Given a <i>full</i> string, compute the positions of each of its
	 * characters within the corresponding <i>lean</i> string.
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string including directional formatting characters.
	 *
	 * @return an array of integers with one element for each of the characters
	 * in the <code>text</code> argument, equal to the offset of the corresponding character 
	 * in the <i>lean</i> string. If there is no corresponding character in the <i>lean</i> string 
	 * (because the specified character is a directional formatting character added when invoking 
	 * {@link #leanToFullText}), the value returned for this character is -1.
	 */
	public int[] fullToLeanMap(STextEnvironment environment, String text) {
		if (procData.processor == null) {
			int[] map = new int[text.length()];
			for (int i = 0; i < map.length; i++)
				map[i] = i;
			return map;
		}
		procData.environment = environment;
		procData.text = text;
		return STextImpl.fullToLeanMap(procData);
	}

	/**
	 * Given a <i>full</i> string, return the offsets of characters
	 * which are directional formatting characters that have been added
	 * in order to ensure correct presentation.
	 * <p>
	 * LRMs (for a string with LTR base direction), RLMs (for a string with RTL base direction) 
	 * are considered as well as leading and trailing LRE, RLE and PDF which might be prefixed 
	 * or suffixed depending on the {@link STextEnvironment#getOrientation orientation} 
	 * of the GUI component used for display.
	 * </p>
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string including directional formatting characters
	 *
	 * @return an array of offsets to the characters in the <code>text</code> argument which 
	 * are directional formatting characters added to ensure correct presentation. The offsets 
	 * are sorted in ascending order.
	 */
	public int[] fullBidiCharOffsets(STextEnvironment environment, String text) {
		if (procData.processor == null)
			return EMPTY_INT_ARRAY;
		procData.environment = environment;
		procData.text = text;
		return STextImpl.fullBidiCharOffsets(procData);
	}

	/**
	 * Get the base direction of a structured text. This base direction may depend on
	 * whether the text contains Arabic or Hebrew words. If the text contains both, 
	 * the first Arabic or Hebrew letter in the text determines which is the governing script.
	 *
	 * @param  environment specifies an environment whose characteristics may affect 
	 * the processor's behavior. If <code>null</code>, the default environment is used.
	 *
	 * @param  text is the structured text string
	 *
	 * @return the base direction of the structured text, {@link #DIR_LTR} or {@link #DIR_RTL}
	 */
	public int getCurDirection(STextEnvironment environment, String text) {
		if (procData.processor == null)
			return DIR_LTR;
		return procData.processor.getDirection(environment, text);
	}

}

Lines 14-22 Link Here

(-)src/org/eclipse/equinox/bidi/STextProcessorFactory.java (-3 / +1 lines)
14	import org.eclipse.equinox.bidi.internal.STextTypesCollector;	14	import org.eclipse.equinox.bidi.internal.STextTypesCollector;
15		15
16	/**	16	/**
17	* This class provides access to registered structured text processors.	17	* This class provides access to registered structured text processors.
18	*
19	* @noinstantiate This class is not intended to be instantiated by clients.
20	*/	18	*/
21	final public class STextProcessorFactory {	19	final public class STextProcessorFactory {
22		20




package org.eclipse.equinox.bidi;

import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.internal.STextImpl;

// TBD experimental
public class STextProcessorMultipass extends STextProcessor {

	//	private int[] state = new int[] {0};

	public STextProcessorMultipass() {

	}
	/*
		public String leanToFullText(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.leanToFullText(processor, environment, text, state);
		}

		public int[] leanToFullMap(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.leanToFullMap(processor, environment, text, state);
		}

		public int[] leanBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.leanBidiCharOffsets(processor, environment, text, state);
		}

		public String fullToLeanText(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.fullToLeanText(processor, environment, text, state);
		}

	public String fullToLeanText(STextProcessor processor, STextEnvironment environment, String text) {
		return STextImpl.fullToLeanText(processor, environment, text, state);
	}

		public int[] fullToLeanMap(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.fullToLeanMap(processor, environment, text, state);
		}

		public int[] fullBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text) {
			return STextImpl.fullBidiCharOffsets(processor, environment, text, state);
		}
	*/
}

Lines 86-92 Link Here

(-)src/org/eclipse/equinox/bidi/STextStringRecord.java (-2 / +2 lines)
86	* @param processor the processor appropriate to handle the type	86	* @param processor the processor appropriate to handle the type
87	* of structured text present in the first segment.	87	* of structured text present in the first segment.
88	* It may be one of the pre-defined processor instances	88	* It may be one of the pre-defined processor instances
89	* appearing in {@link STextEngine}, or it may be an instance	89	* appearing in {@link STextHandler}, or it may be an instance
90	* created by a plug-in or by the application.	90	* created by a plug-in or by the application.
91	*	91	*
92	* @param start offset in the string of the starting character of the first	92	* @param start offset in the string of the starting character of the first
Lines 144-150 Link Here
144	* @param processor the processor appropriate to handle the type	144	* @param processor the processor appropriate to handle the type
145	* of structured text present in this segment.	145	* of structured text present in this segment.
146	* It may be one of the pre-defined processor instances	146	* It may be one of the pre-defined processor instances
147	* appearing in {@link STextEngine}, or it may be an instance	147	* appearing in {@link STextHandler}, or it may be an instance
148	* created by a plug-in or by the application.	148	* created by a plug-in or by the application.
149	*	149	*
150	* @param start offset in the string of the starting character of the	150	* @param start offset in the string of the starting character of the




 *  This class provides a number of convenience functions facilitating the
 *  processing of structured text.
 *
 *  @noextend This class is not intended to be subclassed by clients.
 *  @noinstantiate This class is not intended to be instantiated by clients.
 *
 *  @author Matitiahu Allouche
 */
public final class STextUtil {

	 * If necessary, leading and trailing directional markers (LRE, RLE and PDF) can 
	 * be added depending on the value of the <code>affix</code> argument.
	 * </p>
	 * @see STextHandler#leanBidiCharOffsets leanBidiCharOffsets
	 * 
	 * @param  text the structured text string
	 * @param  offsets an array of offsets to characters in <code>text</code>

	 *         The array must be sorted in ascending order without duplicates.
	 *         This argument may be <code>null</code> if there are no marks to add.
	 * @param  direction the base direction of the structured text.
	 *         It must be one of the values {@link STextHandler#DIR_LTR}, or
	 *         {@link STextHandler#DIR_RTL}.
	 * @param  affix specifies if a prefix and a suffix should be added to
	 *         the result
	 * @return a string corresponding to the source <code>text</code> with

		String curPrefix, curSuffix, full;
		char curMark, c;
		char[] fullChars;
		if (direction == STextHandler.DIR_LTR) {
			curMark = LRM;
			curPrefix = "\u202a\u200e"; /* LRE+LRM *///$NON-NLS-1$
			curSuffix = "\u200e\u202c"; /* LRM+PDF *///$NON-NLS-1$

			separators = defaultSeparators;

		// make sure that LRE/PDF are added around the string
		STextHandler handler = new STextHandler(new STextProcessor(separators));
		return handler.leanToFullText(env, str);
	}

	/**

		STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);
		if (!env.isProcessingNeeded())
			return str;
		STextHandler handler = new STextHandler(processor);
		return handler.leanToFullText(env, str);
	}

	/**

		STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);
		if (!env.isProcessingNeeded())
			return str;
		STextHandler handler = new STextHandler(processor);
		return handler.fullToLeanText(env, str);
	}

}

Lines 43-48 Link Here

(-)src/org/eclipse/equinox/bidi/custom/STextCharTypes.java (-12 / +33 lines)
43	// current orientation	43	// current orientation
44	private int orientation = -1; // "-1" means "unknown"	44	private int orientation = -1; // "-1" means "unknown"
45		45
		46	/**
		47	* Constructor
		48	*
		49	* @param text is the text whose characters are analyzed.
		50	*/
46	public STextCharTypes(String text) {	51	public STextCharTypes(String text) {
47	this.text = text;	52	this.text = text;
48	dirProps = new byte[text.length()];	53	dirProps = new byte[text.length()];
Lines 57-70 Link Here
57	}	62	}
58		63
59	/**	64	/**
60	* @param dirProp bidirectional class of the character. It must be	65	* Get the orientation of the component in which the text will
61	* one of the values which can be returned by	66	* be displayed.
62	* <code>java.lang.Character.getDirectionality</code>.	67	*
		68	* @param environment
		69	*
		70	* @return the orientation as either
		71	* {@link STextEnvironment#ORIENT_LTR} or
		72	* {@link STextEnvironment#ORIENT_RTL}.
63	*/	73	*/
64	public void setBidiTypeAt(int i, byte dirProp) {
65	dirProps[i] = (byte) (dirProp + DIRPROPS_ADD);
66	}
67
68	public int getOrientation(STextEnvironment environment) {	74	public int getOrientation(STextEnvironment environment) {
69	int result;	75	int result;
70	int orient = environment.getOrientation();	76	int orient = environment.getOrientation();
Lines 98-108 Link Here
98	}	104	}
99		105
100	/**	106	/**
101	* Returns directionality of the character in the original string at	107	* Returns directionality of the character in the original string at
102	* the specified index.	108	* the specified index.
103	* @param index position of the character in the <i>lean</i> text	109	*
104	* @return the bidirectional class of the character. It is one of the	110	* @param index position of the character in the <i>lean</i> text
105	* values which can be returned by {@link Character#getDirectionality(char)}	111	*
		112	* @return the bidi type of the character. It is one of the
		113	* values which can be returned by
		114	* {@link Character#getDirectionality(char)}.
106	*/	115	*/
107	public byte getBidiTypeAt(int index) {	116	public byte getBidiTypeAt(int index) {
108	if (hasCachedDirectionAt(index))	117	if (hasCachedDirectionAt(index))
Lines 115-118 Link Here
115	return dirProp;	124	return dirProp;
116	}	125	}
117		126
		127	/**
		128	* Force a bidi type on a character.
		129	*
		130	* @param index is the index of the character whose bidi type is set.
		131	*
		132	* @param dirProp bidirectional type of the character. It must be
		133	* one of the values which can be returned by
		134	* <code>java.lang.Character.getDirectionality</code>.
		135	*/
		136	public void setBidiTypeAt(int index, byte dirProp) {
		137	dirProps[index] = (byte) (dirProp + DIRPROPS_ADD);
		138	}
118	}	139	}




package org.eclipse.equinox.bidi.custom;

public class STextOffsets {
	static final byte L = Character.DIRECTIONALITY_LEFT_TO_RIGHT;
	static final byte R = Character.DIRECTIONALITY_RIGHT_TO_LEFT;
	static final byte AL = Character.DIRECTIONALITY_RIGHT_TO_LEFT_ARABIC;
	static final byte AN = Character.DIRECTIONALITY_ARABIC_NUMBER;
	static final byte EN = Character.DIRECTIONALITY_EUROPEAN_NUMBER;
	static final byte[] STRONGS = {L, R};
	static final int OFFSET_SIZE = 20;
	int[] offsets = new int[OFFSET_SIZE];
	int count; // number of used entries		

	/**
	 *  @return the number of used entries in the offsets array.
	 */
	public int getCount() {
		return count;
	}

	/**
	 *  Mark that all entries in the offsets array are unused.
	 */
	public void resetCount() {
		count = 0;
	}

	/**
	 *  Get the value of a specified entry in the offsets array.
	 *  
	 *  @param  index is the index of the entry of interest.
	 *  
	 *  @return the value of the specified entry.
	 */
	public int getOffset(int index) {
		return offsets[index];
	}

	/**
	 *  Insert an offset value in the offset array so that the array 
	 *  stays in ascending order.
	 *  
	 *  @param  procData is a group of data accessible to processors.
	 *  
	 *  @param  offset is the value to insert.
	 */
	public void insertOffset(STextProcessorData procData, int offset) {
		int index = count - 1; // index of greatest member <= offset
		// look up after which member the new offset should be inserted
		while (index >= 0) {
			int wrkOffset = offsets[index];
			if (offset > wrkOffset)
				break;
			if (offset == wrkOffset)
				return; // avoid duplicates
			index--;
		}
		index++; // index now points at where to insert
		int length = count - index; // number of members to move up
		if (length > 0) // shift right all members greater than offset
			System.arraycopy(offsets, index, offsets, index + 1, length);
		offsets[index] = offset;
		count++; // number of used entries
		// if the offset is 0, adding a mark does not change anything
		if (offset < 1)
			return;
		STextCharTypes dirProps = procData.dirProps;
		if (dirProps == null)
			return;

		byte dirProp = dirProps.getBidiTypeAt(offset);
		// if the current char is a strong one or a digit, we change the
		//   dirProp of the previous char to account for the inserted mark.
		if (dirProp == L || dirProp == R || dirProp == AL || dirProp == EN || dirProp == AN)
			index = offset - 1;
		else
			// if the current char is a neutral, we change its own dirProp
			index = offset;

		dirProps.setBidiTypeAt(index, STRONGS[procData.direction]);
		return;

	}

	/**
	 *  Make sure that there is at least 3 free entries in the offsets array.
	 */
	public void ensureRoom() {
		// make sure there are at least 3 empty slots in offsets
		if ((offsets.length - count) < 3) {
			int[] newOffsets = new int[offsets.length * 2];
			System.arraycopy(offsets, 0, newOffsets, 0, count);
			offsets = newOffsets;
		}
	}

	/**
	 *  Get all and only the used offset entries.
	 *  
	 *  @return the current used entries of the offsets array.
	 */
	public int[] getArray() {
		if (count == offsets.length)
			return offsets;
		int[] array = new int[count];
		System.arraycopy(offsets, 0, array, 0, count);
		return array;
	}
}




 ******************************************************************************/
package org.eclipse.equinox.bidi.custom;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.internal.STextImpl;

/**

 *  <ul>
 *    <li>Processor instances may be accessed simultaneously by
 *        several threads. They should have no instance variables.</li>
 *    <li>All the user methods in {@link STextHandler} implement a logic
 *        common to all processors, located in {@link STextImpl}.
 *        These methods all have a first argument which is a processor
 *        instance.
 *        The common logic uses processor methods to query the
 *        characteristics of the specific processor:
 *        <ul>

 *        method, the common logic will repeatedly invoke the processor's
 *        {@link #indexOfSpecial indexOfSpecial} method to let it signal the
 *        presence of special strings which may further delimit the source text.</li>
 *    <li>When such a special case is signaled by the processor, the common
 *        logic will call the processor's {@link #processSpecial processSpecial}
 *        method to give it the opportunity to handle it as needed. Typical
 *        actions that the processor may perform are to add directional marks
 *        unconditionally (by calling {@link #insertMark insertMark} or
 *        conditionally (by calling {@link #processSeparator processSeparator}).</li>
 *  </ul>
 *
 * @see STextHandler
 * @author Matitiahu Allouche
 */
public class STextProcessor {

	final private String separators;

	/**
	 *  Creates a new instance of the STextProcessor class.
	 */
	public STextProcessor() {
		separators = ""; //$NON-NLS-1$
	}

	/**
	 *  Creates a new instance of the STextProcessor class.
	 *  
	 *  @param  separators string consisting of characters that split 
	 *          the text into fragments
	 */
	public STextProcessor(String separators) {
		this.separators = separators;
	}

	/**
	 *  Locate occurrences of special strings within a structured text
	 *  and return their indexes one after the other in successive calls.
	 *  <p>
	 *  This method is called repeatedly from the code implementing
	 *  {@link STextHandler#leanToFullText} if the number of special cases 
	 *  returned by {@link #getSpecialsCount} is greater than zero.
	 *  </p><p>
	 *  A processor handling special cases must override this method.
	 *  </p>
	 *  @param  procData is a group of data accessible by the processor.
	 *         the processor. This parameter may be specified as
	 *         <code>null</code>, in which case the
	 *         {@link STextEnvironment#DEFAULT DEFAULT}
	 *         environment should be assumed.
	 *
	 *  @param  caseNumber is the number of the special case to locate.
	 *          This number varies from 1 to the number of special cases
	 *          returned by {@link #getSpecialsCount getSpecialsCount}
	 *          for this processor.
	 *          The meaning of this number is internal to the class
	 *          implementing <code>indexOfSpecial</code>.
	 *
	 * @param  offsets is a parameter received by <code>indexOfSpecial</code>
	 *         uniquely to be used as argument for calls to methods which
	 *         need it.
	 *
	 * @param  caseNumber number of the special case to locate.
	 *         This number varies from 1 to the number of special cases
	 *         returned by {@link #getSpecialsCount getSpecialsCount}
	 *         for this processor.
	 *         The meaning of this number is internal to the class
	 *         implementing <code>indexOfSpecial</code>.
	 *
	 *  @param  fromIndex is the index within <code>text</code> to start
	 *          the search from.
	 *
	 *  @return the position where the start of the special case
	 *          corresponding to <code>caseNumber</code> was located.
	 *          The method must return the first occurrence of whatever
	 *          identifies the start of the special case starting from
	 *          <code>fromIndex</code>. The method does not have to check if
	 *          this occurrence appears within the scope of another special
	 *          case (e.g. a comment starting delimiter within the scope of
	 *          a literal or vice-versa).
	 *          <br>If no occurrence is found, the method must return -1.
	 *
	 * @throws  IllegalStateException If not overridden, this method throws an
	 *          <code>IllegalStateException</code>. This is appropriate behavior
	 *          (and does not need to be overridden) for processors whose
	 *          number of special cases is zero, which means that
	 *          <code>indexOfSpecial</code> should never be called for them.
	 */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		// This method must be overridden by all subclasses with special cases.
		throw new IllegalStateException("A processor with specialsCount > 0 must have an indexOfSpecial() method."); //$NON-NLS-1$
	}

	/**
	 *  This method handles special cases specific to this processor.
	 *  It is called by {@link STextHandler#leanToFullText} when a special case occurrence 
	 *  is located by {@link #indexOfSpecial}.
	 *  <p>
	 *  If a special processing cannot be completed within a current call to
	 *  <code>processSpecial</code> (for instance, a comment has been started
	 *  in the current line but its end appears in a following line),
	 *  <code>processSpecial</code> should specify a final <b>state</b> by
	 *  calling {@link STextHandler#setState}.
	 *  The meaning of this <b>state</b> is internal to the processor.
	 *  On a later call to {@link STextHandler#leanToFullText},
	 *  <code>processSpecial</code> will be called with that value 
	 *  for parameter <code>caseNumber</code> and <code>-1</code> for parameter 
	 *  <code>separLocation</code> and should perform whatever initializations are required
	 *  depending on the <b>state</b>.
	 *  </p><p>
	 *  A processor handling special cases (with a number of
	 *  special cases greater than zero) must override this method.
	 *  </p>
	 *  @param  procData is a group of data accessible to the processor.
	 * @param  environment the current environment, which may affect the behavior of
	 *         the processor. This parameter may be specified as
	 *         <code>null</code>, in which case the
	 *         {@link STextEnvironment#DEFAULT DEFAULT}
	 *         environment should be assumed.
	 *
	 * @param  text is the structured text string before
	 *         addition of any directional formatting characters.
	 *
	 * @param  dirProps is a parameter received by <code>processSpecial</code>
	 *         uniquely to be used as argument for calls to methods which
	 *         need it.
	 *
	 * @param  offsets is a parameter received by <code>processSpecial</code>
	 *         uniquely to be used as argument for calls to methods which
	 *         need it.
	 *
	 * @param  state is an integer array with at least one element.
	 *         If the processor needs to signal the occurrence of a
	 *         special case which must be passed to the next call to
	 *         <code>leanToFullText</code> (for instance, a comment or a
	 *         literal started but not closed in the current
	 *         <code>text</code>), it must put a value in the first element
	 *         of the <code>state</code> parameter.
	 *         This number must be >= 1 and less or equal to the number of special
	 *         cases returned by {@link #getSpecialsCount getSpecialsCount}
	 *         by this processor.
	 *         This number is passed back to the caller
	 *         and should be specified as <code>state</code> argument
	 *         in the next call to <code>leanToFullText</code> together
	 *         with the continuation text.
	 *         The meaning of this number is internal to the processor.
	 *
	 *  @param  caseNumber number of the special case to handle.
	 *
	 *  @param  separLocation the position returned by
	 *          {@link #indexOfSpecial indexOfSpecial}. In calls to
	 *          {@link STextHandler#leanToFullText leanToFullText} and other
	 *          methods of {@link STextHandler} setting a  non-null
	 *          <b>state</b> value, <code>processSpecial</code> is
	 *          called when initializing the processing with the value of
	 *          <code>caseNumber</code> equal to the value returned
	 *          as <b>state</b> and the value of
	 *          <code>separLocation</code> equal to <code>-1</code>.
	 *
	 *  @return the position after the scope of the special case ends.
	 *          For instance, the position after the end of a comment,
	 *          the position after the end of a literal.
	 *          <br>A value greater or equal to the length of <code>text</code>
	 *          means that there is no further occurrence of this case in the
	 *          current structured text.
	 *
	 *  @throws IllegalStateException If not overridden, this method throws an
	 *          <code>IllegalStateException</code>. This is appropriate behavior
	 *          (and does not need to be overridden) for processors whose
	 *          number of special cases is zero, which means that
	 *          <code>processSpecial</code> should never be called for them.
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		// This method must be overridden by all subclasses with any special case.
		throw new IllegalStateException("A processor with specialsCount > 0 must have a processSpecial() method."); //$NON-NLS-1$
	}

	/**
	 *  This method can be called from within {@link #indexOfSpecial} or
	 *  {@link #processSpecial} in extensions of <code>STextProcessor</code>
	 *  to specify that a mark character must be added before the character
	 *  at the specified position of the <i>lean</i> text when generating the
	 *  <i>full</i> text. The mark character will be LRM for structured text
	 *  with a LTR base direction, and RLM for structured text with RTL
	 *  base direction. The mark character is not added physically by this
	 *  method, but its position is noted and will be used when generating
	 *  the <i>full</i> text.
	 *
	 *  @param  procData is a group of data accessible to the processor.
	 *         parameter to <code>indexOfSpecial</code> or
	 *         <code>processSpecial</code>.
	 *
	 *  @param  offset position of the character in the <i>lean</i> text.
	 *          It must be a non-negative number smaller than the length
	 *          of the <i>lean</i> text.
	 *          For the benefit of efficiency, it is better to insert
	 *          multiple marks in ascending order of the offsets.
	 * @param  offsets is a parameter received by <code>indexOfSpecial</code>
	 *         or <code>processSpecial</code>, uniquely to be used as argument
	 *         for calls to <code>insertMark</code> and other methods used
	 *         by processors.
	 *
	 * @param  offset position of the character in the <i>lean</i> text.
	 *         It must be a non-negative number smaller than the length
	 *         of the <i>lean</i> text.
	 *         For the benefit of efficiency, it is better to insert
	 *         multiple marks in ascending order of the offsets.
	 */
	public static final void insertMark(STextProcessorData procData, int offset) {
		procData.offsets.insertOffset(procData, offset);
	}

	/**
	 *  This method can be called from within {@link #indexOfSpecial} or
	 *  {@link #processSpecial} in extensions of <code>STextProcessor</code> to add 
	 *  a directional mark before a separator if needed for correct display, 
	 *  depending on the base direction of the text and on the type of the
	 *  characters in the <i>lean</i> text preceding and following the separator itself.
	 *  <p>
	 *  The logic implemented in this method considers the text before
	 *  <code>separLocation</code> and the text following it. If, and only if,
	 *  a directional mark is needed to insure that the two parts of text
	 *  will be laid out according to the base direction, a mark will be
	 *  added when generating the <i>full</i> text.
	 *  </p>
	 *  @param  procData is a group of data accessible to the processor.
	 *         parameter to <code>indexOfSpecial</code> or
	 *         <code>processSpecial</code>.
	 *
	 * @param  dirProps is a parameter received by <code>indexOfSpecial</code>
	 *         or <code>processSpecial</code>, uniquely to be used as argument
	 *         for calls to <code>processSeparator</code> and other methods used
	 *         by processors.
	 *
	 * @param  offsets is a parameter received by <code>indexOfSpecial</code>
	 *         or <code>processSpecial</code>, uniquely to be used as argument
	 *         for calls to <code>processSeparator</code> and other methods used
	 *         by processors.
	 *
	 *  @param  separLocation offset of the separator in the <i>lean</i> text.
	 *          It must be a non-negative number smaller than the length
	 *          of the <i>lean</i> text.
	 */
	public static final void processSeparator(STextProcessorData procData, int separLocation) {
		STextImpl.processSeparator(procData, separLocation);
	}

	/**
	 *  Indicate the separators to use for the current processor.
	 *  This method is invoked before starting the processing.
	 *  <p>
	 *  If no separators are specified, this method returns an empty string.
	 *  </p>
	 *  @param  environment the current environment, which may affect the behavior of
	 *          the processor. This parameter may be specified as
	 *          <code>null</code>, in which case the
	 *          {@link STextEnvironment#DEFAULT DEFAULT}
	 *          environment should be assumed.
	 *
	 *  @return a string grouping one-character separators which separate
	 *          the structured text into tokens.
	 */
	public String getSeparators(STextEnvironment environment) {
		return separators;
	}

	/**
	 *  Indicate the base text direction appropriate for an instance of structured text.
	 *  This method is invoked before starting the processing.
	 *  <p>
	 *  If not overridden, this method returns <code>DIR_LTR</code>.
	 *  </p>
	 *  @param  environment the current environment, which may affect the behavior of
	 *          the processor. This parameter may be specified as
	 *          <code>null</code>, in which case the
	 *          {@link STextEnvironment#DEFAULT DEFAULT}
	 *          environment should be assumed.
	 *
	 *  @param  text is the structured text string to process.
	 *
	 *  @return the base direction of the structured text. This direction
	 *          may not be the same depending on the environment and on
	 *          whether the structured text contains Arabic or Hebrew
	 *          letters.<br>
	 *          The value returned is either
	 *          {@link STextHandler#DIR_LTR DIR_LTR} or {@link STextHandler#DIR_RTL DIR_RTL}.
	 */
	public int getDirection(STextEnvironment environment, String text) {
		return STextHandler.DIR_LTR;
	}

	/**
	 *  Indicate the base text direction appropriate for an instance of structured text.
	 *  This method is invoked before starting the processing.
	 *  <p>
	 *  If not overridden, this method returns <code>DIR_LTR</code>.
	 *  </p>
	 *  @param  environment the current environment, which may affect the behavior of
	 *          the processor. This parameter may be specified as
	 *          <code>null</code>, in which case the
	 *          {@link STextEnvironment#DEFAULT DEFAULT}
	 *          environment should be assumed.
	 *
	 *  @param  text is the structured text string to process.
	 *
	 *  @param  dirProps is a parameter received uniquely to be used as argument
	 *          for calls to {@link STextCharTypes#getBidiTypeAt}
	 *          and other methods used by processors.
	 *
	 *  @return the base direction of the structured text. This direction
	 *          may not be the same depending on the environment and on
	 *          whether the structured text contains Arabic or Hebrew
	 *          letters.<br>
	 *          The value returned is either
	 *          {@link STextHandler#DIR_LTR DIR_LTR} or {@link STextHandler#DIR_RTL DIR_RTL}.
	 */
	public int getDirection(STextEnvironment environment, String text, STextCharTypes dirProps) {
		return STextHandler.DIR_LTR;
	}

	/**
	 *  Indicate the number of special cases handled by the current processor.
	 *  This method is invoked before starting the processing.
	 *  If the number returned is zero, {@link #indexOfSpecial} and
	 *  {@link #processSpecial} will not be invoked.
	 *  <p>
	 *  If not overridden, this method returns <code>zero</code>.
	 *  </p>
	 *  @param  environment the current environment, which may affect the behavior of
	 *          the processor. This parameter may be specified as
	 *          <code>null</code>, in which case the
	 *          {@link STextEnvironment#DEFAULT DEFAULT}
	 *          environment should be assumed.
	 *
	 *  @return the number of special cases for the associated processor.
	 *          Special cases exist for some types of structured text
	 *          processors. They are implemented by overriding methods
	 *          {@link STextProcessor#indexOfSpecial} and {@link STextProcessor#processSpecial}.
	 *          Examples of special cases are comments, literals, or
	 *          anything which is not identified by a one-character separator.
	 *
	 */
	public int getSpecialsCount(STextEnvironment environment) {

	}

	/**
	 *  Checks if there is a need for processing structured text.
	 *  This method is invoked before starting the processing. If the
	 *  processor returns <code>true</code>, no directional formatting
	 *  characters are added to the <i>lean</i> text and the processing
	 *  is shortened.
	 *  <p>
	 *  If not overridden, this method returns <code>false</code>.
	 *  </p>

	 *  @return a flag indicating if there is no need to process the structured
	 *          text to add directional formatting characters.
	 *         {@link STextEnvironment#DEFAULT DEFAULT}
	 *         environment should be assumed.
	 *
	 * @param  text is the structured text string to process.
	 *
	 * @param  dirProps is a parameter received uniquely to be used as argument
	 *         for calls to <code>getDirProp</code> and other methods used
	 *         by processors.
	 *
	 * @return a flag indicating if there is no need to process the structured
	 *         text to add directional formatting characters.
	 *
	 */
	public boolean skipProcessing(STextProcessorData procData) {
		return false;
	}

}




package org.eclipse.equinox.bidi.custom;

import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;

/**
 *  This class regroups core information related to structured text operations.
 *  It is convenient to pass information between caller and called method.
 *  
 *  @author Matitiahu Allouche
 *
 */
public class STextProcessorData {
	/**
	 *  The handler 
	 */
	public STextHandler handler;
	/**
	 *  The processor
	 */
	public STextProcessor processor;
	/**
	 *  The environment
	 */
	public STextEnvironment environment;
	/**
	 *  The class instance to get and set the bidi type of characters
	 */
	public STextCharTypes dirProps;
	/**
	 *  The class instance to manipulate the array of offsets to 
	 *  directional control characters
	 */
	public STextOffsets offsets;
	/**
	 *  The source text
	 */
	public String text;
	/**
	 *  The base direction of the source text
	 */
	public int direction;
	/**
	 *  The orientation of the component which is to display the text
	 */
	public int orientation;
	/**
	 *  The length of the prefix (LRE, RLE, LRM or RLM) to be prepended 
	 *  to the text
	 */
	public int prefixLength;
}




<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<META name="Author" content="Matitiahu Allouche">
</head>
<body bgcolor="white">

This package provides classes for
developing structured text processors.

<ul>
  <li>{@link <a href="STextProcessor.html">STextProcessor</a>}
      is a generic processor to be used as superclass for specific
      processors.</li>
  <li>{@link <a href="STextProcessorData.html">STextProcessorData</a>}
      regroups data which can be used by processors.</li>
  <li>{@link <a href="STextCharTypes.html">STextStringCharTypes</a>}
      allows getting and setting the bidi class of characters.</li>
  <li>{@link <a href="STextOffsets.html">STextStringOffsets</a>}
      allows manipulation of offsets at which directional control
      characters are inserted.</li>
</ul>

This package is to be used together with package
{@link <a href="..\package-summary.html">
org.eclipse.equinox.bidi</a>}.

</body>
</html>




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal;

import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.custom.STextCharTypes;
import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.custom.STextProcessorData;

/**
 *  A base processor for structured text composed of text segments separated 

	 *
	 *  @see #getDelimiters
	 */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		char delim = getDelimiters().charAt((caseNumber - 1) * 2);
		return procData.text.indexOf(delim, fromIndex);
	}

	/**

	 *  @return the position after the matching end delimiter, or the length
	 *          of <code>text</code> if no end delimiter is found.
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		STextProcessor.processSeparator(procData, separLocation);
		int loc = separLocation + 1;
		char delim = getDelimiters().charAt((caseNumber * 2) - 1);
		loc = procData.text.indexOf(delim, loc);
		if (loc < 0)
			return procData.text.length();
		return loc + 1;
	}





 ******************************************************************************/
package org.eclipse.equinox.bidi.internal;

import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.custom.STextCharTypes;
import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.custom.STextProcessorData;

/**
 *  A base processor for structured text composed of text segments separated 

	 *  and skips until after the matching end delimiter,
	 *  ignoring possibly escaped end delimiters.
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		STextProcessor.processSeparator(procData, separLocation);
		int location = separLocation + 1;
		char delim = getDelimiters().charAt((caseNumber * 2) - 1);
		String text = procData.text;
		while (true) {
			location = text.indexOf(delim, location);
			if (location < 0)




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.*;

/**
 *  <code>STextImpl</code> provides the code which implements the API in
 *  {@link STextHandler}. All its public methods are shadows of similarly
 *  signed methods of <code>STextHandler</code>, and their documentation
 *  is by reference to the methods in <code>STextHandler</code>.
 *
 *  @author Matitiahu Allouche
 */

	static final char PDF = 0x202C;
	static final char[] MARKS = {LRM, RLM};
	static final char[] EMBEDS = {LRE, RLE};
	static final byte[] STRONGS = {L, R};
	static final int PREFIX_LENGTH = 2;
	static final int SUFFIX_LENGTH = 2;
	static final int FIXES_LENGTH = PREFIX_LENGTH + SUFFIX_LENGTH;

	static final STextEnvironment IGNORE_ENVIRONMENT = new STextEnvironment(null, false, STextEnvironment.ORIENT_IGNORE);

	/**
	 *  Prevent creation of a STextHandler instance
	 */
	private STextImpl() {
		// nothing to do
	}

	static long computeNextLocation(STextProcessorData procData, int[] locations, int curPos) {
		STextProcessor processor = procData.processor;
		String separators = processor.getSeparators(procData.environment);
		int separCount = separators.length();
		int specialsCount = processor.getSpecialsCount(procData.environment);
		String text = procData.text;
		int len = text.length();
		int nextLocation = len;
		int idxLocation = 0;

		for (int i = 0; i < specialsCount; i++) {
			int location = locations[separCount + i];
			if (location < curPos) {
				procData.offsets.ensureRoom();
				location = processor.indexOfSpecial(procData, i + 1, curPos);
				if (location < 0)
					location = len;
				locations[separCount + i] = location;

	/**
	 *  @see STextProcessor#processSeparator STextProcessor.processSeparator
	 */
	public static void processSeparator(STextProcessorData procData, int separLocation) {
		String text = procData.text;
		int len = text.length();
		STextCharTypes dirProps = procData.dirProps;
		STextOffsets offsets = procData.offsets;
		if (procData.direction == STextHandler.DIR_RTL) {
			// the structured text base direction is RTL
			for (int i = separLocation - 1; i >= 0; i--) {
				byte dirProp = dirProps.getBidiTypeAt(i);

						if (dirProp == R || dirProp == AL)
							return;
						if (dirProp == L || dirProp == EN) {
							offsets.insertOffset(procData, separLocation);
							return;
						}
					}

					if (dirProp == L)
						return;
					if (dirProp == R || dirProp == EN || dirProp == AL || dirProp == AN) {
						offsets.insertOffset(procData, separLocation);
						return;
					}
				}

					if (dirProp == L)
						return;
					if (dirProp == AL || dirProp == AN || dirProp == R) {
						offsets.insertOffset(procData, separLocation);
						return;
					}
				}

	/**
	 *  When the orientation is <code>ORIENT_LTR</code> and the
	 *  structured text has a RTL base direction,
	 *  {@link STextHandler#leanToFullText leanToFullText}
	 *  adds RLE+RLM at the head of the <i>full</i> text and RLM+PDF at its
	 *  end.
	 *  <p>
	 *  When the orientation is <code>ORIENT_RTL</code> and the
	 *  structured text has a LTR base direction,
	 *  {@link STextHandler#leanToFullText leanToFullText}
	 *  adds LRE+LRM at the head of the <i>full</i> text and LRM+PDF at its
	 *  end.
	 *  <p>
	 *  When the orientation is <code>ORIENT_CONTEXTUAL_LTR</code> or
	 *  <code>ORIENT_CONTEXTUAL_RTL</code> and the data content would resolve
	 *  to a RTL orientation while the structured text has a LTR base
	 *  direction, {@link STextHandler#leanToFullText leanToFullText}
	 *  adds LRM at the head of the <i>full</i> text.
	 *  <p>
	 *  When the orientation is <code>ORIENT_CONTEXTUAL_LTR</code> or
	 *  <code>ORIENT_CONTEXTUAL_RTL</code> and the data content would resolve
	 *  to a LTR orientation while the structured text has a RTL base
	 *  direction, {@link STextHandler#leanToFullText leanToFullText}
	 *  adds RLM at the head of the <i>full</i> text.
	 *  <p>
	 *  When the orientation is <code>ORIENT_UNKNOWN</code> and the
	 *  structured text has a LTR base direction,
	 *  {@link STextHandler#leanToFullText leanToFullText}
	 *  adds LRE+LRM at the head of the <i>full</i> text and LRM+PDF at its
	 *  end.
	 *  <p>
	 *  When the orientation is <code>ORIENT_UNKNOWN</code> and the
	 *  structured text has a RTL base direction,
	 *  {@link STextHandler#leanToFullText leanToFullText}
	 *  adds RLE+RLM at the head of the <i>full</i> text and RLM+PDF at its
	 *  end.
	 *  <p>
	 *  When the orientation is <code>ORIENT_IGNORE</code>,
	 *  {@link STextHandler#leanToFullText leanToFullText} does not add any directional
	 *  formatting characters as either prefix or suffix of the <i>full</i> text.
	 *  <p>
	 *  @see STextHandler#leanToFullText STextHandler.leanToFullText
	 */
	public static String leanToFullText(STextProcessorData procData) {
		String text = procData.text;
		int len = text.length();
		if (len == 0)
			return text;
		leanToFullCommon(procData);
		STextOffsets offsets = procData.offsets;
		int count = offsets.getCount();
		if (count == 0 && procData.prefixLength == 0)
		if (count == 0 && prefixLength == 0)
			return text;
		int newLen = len + count;
		if (procData.prefixLength == 1)
			newLen++; /* +1 for a mark char */
		else if (procData.prefixLength == 2)
			newLen += FIXES_LENGTH;
		char[] fullChars = new char[newLen];
		int added = procData.prefixLength;
		// add marks at offsets
		char curMark = MARKS[procData.direction];
		for (int i = 0, j = 0; i < len; i++) {
		for (int i = 0, j = OFFSETS_SHIFT; i < len; i++) {
			char c = text.charAt(i);
			if (j < count && i == offsets.getOffset(j)) {
			if (j < offsets[0] && i == offsets[j]) {
				fullChars[i + added] = curMark;
				added++;
				j++;
			}
			fullChars[i + added] = c;
		}
		if (procData.prefixLength > 0) { /* add prefix/suffix ? */
			if (procData.prefixLength == 1) { /* contextual orientation */
				fullChars[0] = curMark;
			} else {
				// When the orientation is RTL, we need to add EMBED at the
				// start of the text and PDF at its end.
				// However, because of a bug in Windows' handling of LRE/PDF,
				// we add EMBED_PREFIX at the start and EMBED_SUFFIX at the end.
				char curEmbed = EMBEDS[procData.direction];
				fullChars[0] = curEmbed;
				fullChars[1] = curMark;
				fullChars[newLen - 1] = PDF;

	}

	/**
	 *  @see STextHandler#leanToFullMap STextHandler.leanToFullMap
	 */
	public static int[] leanToFullMap(STextProcessorData procData) {
		String text = procData.text;
		int len = text.length();
		if (len == 0)
			return EMPTY_INT_ARRAY;
		leanToFullCommon(procData);
		STextOffsets offsets = procData.offsets;
		int prefixLength = offsets[1];
		int[] map = new int[len];
		int count = procData.offsets.getCount();
		int added = procData.prefixLength;
		for (int pos = 0, i = 0; pos < len; pos++) {
			if (i < count && pos == offsets.getOffset(i)) {
				added++;
				i++;
			}

	}

	/**
	 *  @see STextHandler#leanBidiCharOffsets STextHandler.leanBidiCharOffsets
	 */
	public static int[] leanBidiCharOffsets(STextProcessorData procData) {
		String text = procData.text;
		int len = text.length();
		if (len == 0)
			return EMPTY_INT_ARRAY;
		leanToFullCommon(procData);
		return procData.offsets.getArray();
		// offsets[0] contains the number of used entries
		int count = offsets[0] - OFFSETS_SHIFT;
		int[] result = new int[count];
		System.arraycopy(offsets, OFFSETS_SHIFT, result, 0, count);
		return result;
	}

	static void leanToFullCommon(STextProcessorData procData) {
		if (procData.environment == null)
			procData.environment = STextEnvironment.DEFAULT;
		String text = procData.text;
			state = new int[1];
			state[0] = STextEngine.STATE_INITIAL;
		}
		int len = text.length();
		STextHandler handler = procData.handler;
		STextProcessor processor = procData.processor;
		STextEnvironment environment = procData.environment;
		STextOffsets offsets = procData.offsets;
		STextCharTypes dirProps = procData.dirProps = new STextCharTypes(text);
		procData.orientation = dirProps.getOrientation(environment);
		procData.direction = processor.getDirection(environment, text, dirProps);
		offsets.resetCount();
		if (!processor.skipProcessing(procData)) {
			// initialize locations
			int separCount = processor.getSeparators(environment).length();
			int[] locations = new int[separCount + processor.getSpecialsCount(environment)];

			}
			// current position
			int curPos = 0;
			int state = handler.getState();
			if (state > STextHandler.STATE_INITIAL) {
				offsets.ensureRoom();
				handler.setState(STextHandler.STATE_INITIAL);
				curPos = processor.processSpecial(procData, state, -1);
			}
			while (true) {
				// location of next token to handle
				int nextLocation;
				// index of next token to handle (if < separCount, this is a separator; otherwise a special case
				int idxLocation;
				long res = computeNextLocation(procData, locations, curPos);
				nextLocation = (int) (res & 0x00000000FFFFFFFF); /* low word */
				if (nextLocation >= len)
					break;
				offsets.ensureRoom();
				idxLocation = (int) (res >> 32); /* high word */
				if (idxLocation < separCount) {
					processSeparator(procData, nextLocation);
					processSeparator(text, dirProps, offsets, nextLocation);
					curPos = nextLocation + 1;
				} else {
					offsets = ensureRoomInOffsets(offsets);
					idxLocation -= (separCount - 1); // because caseNumber starts from 1
					curPos = processor.processSpecial(procData, idxLocation, nextLocation);
				}
				if (curPos >= len)
					break;
			} // end while
		} // end if (!processor.skipProcessing())
		if (procData.orientation == STextEnvironment.ORIENT_IGNORE)
			procData.prefixLength = 0;
		else {
			// recompute orient since it may have changed if contextual
			procData.orientation = dirProps.getOrientation(environment);
			if (procData.orientation == procData.direction && procData.orientation != STextEnvironment.ORIENT_UNKNOWN)
				procData.prefixLength = 0;
			else if ((environment.getOrientation() & STextEnvironment.ORIENT_CONTEXTUAL_LTR) != 0)
				procData.prefixLength = 1;
			else
				procData.prefixLength = 2;
		}
		return offsets;
	}

	/**
	 *  @see STextHandler#fullToLeanText STextHandler.fullToLeanText
	 */
	public static String fullToLeanText(STextProcessorData procData) {
		String text = procData.text;
		if (text.length() == 0)
			return text;
		if (procData.environment == null)
			procData.environment = STextEnvironment.DEFAULT;
		int dir = procData.processor.getDirection(procData.environment, text);
			state = new int[1];
			state[0] = STextEngine.STATE_INITIAL;
		}
		int dir = processor.getDirection(environment, text);
		char curMark = MARKS[dir];
		char curEmbed = EMBEDS[dir];
		int i; // used as loop index

				chars[i - cnt] = c;
		}
		String lean = new String(chars, 0, lenText - cnt);
		String full = procData.handler.leanToFullText(IGNORE_ENVIRONMENT, lean);
		if (full.equals(text))
			return lean;


	}

	/**
	 *  @see STextHandler#fullToLeanMap STextHandler.fullToLeanMap
	 */
	public static int[] fullToLeanMap(STextProcessorData procData) {
		String full = procData.text;
		int lenFull = full.length();
		if (lenFull == 0)
			return EMPTY_INT_ARRAY;
		String lean = procData.handler.fullToLeanText(procData.environment, full);
		int lenLean = lean.length();
		int dir = procData.processor.getDirection(procData.environment, lean);
		char curMark = MARKS[dir];
		char curEmbed = EMBEDS[dir];
		int[] map = new int[lenFull];

	}

	/**
	 *  @see STextHandler#fullBidiCharOffsets STextHandler.fullBidiCharOffsets
	 */
	public static int[] fullBidiCharOffsets(STextProcessorData procData) {
		String full = procData.text;
		int lenFull = full.length();
		if (lenFull == 0)
			return EMPTY_INT_ARRAY;
		String lean = procData.handler.fullToLeanText(procData.environment, full);
		procData.text = full;
		procData.dirProps = new STextCharTypes(full);
		STextOffsets offsets = procData.offsets;
		offsets.resetCount();
		int lenLean = lean.length();
		int idxLean, idxFull;
		// lean must be a subset of Full, so we only check on iLean < leanLen

			if (full.charAt(idxFull) == lean.charAt(idxLean))
				idxLean++;
			else {
				offsets.ensureRoom();
				offsets.insertOffset(procData, idxFull);
			}
		}
		for (; idxFull < lenFull; idxFull++) {
			offsets.ensureRoom();
			offsets.insertOffset(procData, idxFull);
		}
		return offsets.getArray();
		System.arraycopy(offsets, OFFSETS_SHIFT, result, 0, result.length);
		return result;
	}

	static int[] ensureRoomInOffsets(int[] offsets) {
		// make sure
		if ((offsets.length - offsets[0]) < 3) {
			int[] newOffsets = new int[offsets.length * 2];
			System.arraycopy(offsets, 0, newOffsets, 0, offsets[0]);
			return newOffsets;
		}
		return offsets;
	}

	/**
	 *  @see STextProcessor#insertMark STextProcessor.insertMark
	 */
	public static void insertMark(String text, STextCharTypes dirProps, int[] offsets, int offset) {
		int count = offsets[0];// number of used entries
		int index = count - 1; // index of greatest member <= offset
		// look up after which member the new offset should be inserted
		while (index >= OFFSETS_SHIFT) {
			int wrkOffset = offsets[index];
			if (offset > wrkOffset)
				break;
			if (offset == wrkOffset)
				return; // avoid duplicates
			index--;
		}
		index++; // index now points at where to insert
		int length = count - index; // number of members to move up
		if (length > 0) // shift right all members greater than offset
			System.arraycopy(offsets, index, offsets, index + 1, length);
		offsets[index] = offset;
		offsets[0]++; // number of used entries
		// if the offset is 0, adding a mark does not change anything
		if (dirProps == null || offset < 1)
			return;

		byte dirProp = dirProps.getBidiTypeAt(offset);
		// if the current char is a strong one or a digit, we change the
		//   dirProp of the previous char to account for the inserted mark.
		if (dirProp == L || dirProp == R || dirProp == AL || dirProp == EN || dirProp == AN)
			index = offset - 1;
		else
			// if the current char is a neutral, we change its own dirProp
			index = offset;

		int dir = offsets[2]; // current structured text direction
		dirProps.setBidiTypeAt(index, STRONGS[dir]);
		return;
	}

}




package org.eclipse.equinox.bidi.internal;

import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.custom.STextCharTypes;
import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.custom.STextProcessorData;

/**
 *  A base processor for structured text composed of two parts separated by a separator.

	 *
	 *  @see #getSeparators getSeparators
	 */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		return procData.text.indexOf(this.getSeparators(procData.environment).charAt(0), fromIndex);
	}

	/**

	 *
	 *  @return the length of <code>text</code>.
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		STextProcessor.processSeparator(procData, separLocation);
		return procData.text.length();
	}

	/**

	public int getSpecialsCount(STextEnvironment environment) {
		return 1;
	}

}




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal.consumable;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.STextCharTypes;
import org.eclipse.equinox.bidi.internal.STextDelimsEsc;


		super("<>.:,;@"); //$NON-NLS-1$
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment
	 *                language) is Arabic.</li>
	 *            <li>The domain part of the email address contains
	 *                at least one RTL character.</li>
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text) {
		return getDirection(environment, text, new STextCharTypes(text));
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment

	 *            <li>The domain part of the email address contains
	 *                at least one RTL character.</li>
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text, STextCharTypes dirProps) {
		if (environment == null)
			environment = STextEnvironment.DEFAULT;
		String language = environment.getLanguage();
		if (!language.equals("ar")) //$NON-NLS-1$
			return STextHandler.DIR_LTR;
		int domainStart;
		domainStart = text.indexOf('@');
		if (domainStart < 0)

		for (int i = domainStart; i < text.length(); i++) {
			byte dirProp = dirProps.getBidiTypeAt(i);
			if (dirProp == AL || dirProp == R)
				return STextHandler.DIR_RTL;
		}
		return STextHandler.DIR_LTR;
	}

	/**

	protected String getDelimiters() {
		return "()\"\""; //$NON-NLS-1$
	}

}




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal.consumable;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.custom.STextProcessorData;
import org.eclipse.equinox.bidi.internal.STextActivator;

/**

 *  <p>
 *  In applications like an editor where parts of the text might be modified
 *  while other parts are not, the user may want to call
 *  {@link STextHandler#leanToFullText leanToFullText}
 *  separately on each line and save the initial <b>state</b> of each line (this is
 *  the final <b>state</b> of the previous line which can be retrieved using
 *  {@link STextHandler#getState}.
 *  If both the content of a line and its initial
 *  <b>state</b> have not changed, the user can be sure that
 *  the last <i>full</i> text computed for this line has not changed either.
 *
 *  @see STextEngine#leanToFullText explanation of state in leanToFullText
 *
 *  @author Matitiahu Allouche
 */

	     *    <li>comments starting with slash-slash</li>
	     *  </ol>
	     */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		String text = procData.text;
		switch (caseNumber) {
			case 1 : /* space */
				return text.indexOf(' ', fromIndex);

	     *    <li>skip until after a line separator</li>
	     *  </ol>
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		int location, counter, i;

		STextProcessor.processSeparator(procData, separLocation);
		String text = procData.text;
		switch (caseNumber) {
			case 1 : /* space */
				separLocation++;
				while (separLocation < text.length() && text.charAt(separLocation) == ' ') {
					procData.dirProps.setBidiTypeAt(separLocation, WS);
					separLocation++;
				}
				return separLocation;

					location = separLocation + 2; // skip the opening slash-aster
				location = text.indexOf("*/", location); //$NON-NLS-1$
				if (location < 0) {
					procData.handler.setState(caseNumber);
					return text.length();
				}
				// we need to call processSeparator since text may follow the
				//  end of comment immediately without even a space
				STextProcessor.processSeparator(procData, location);
				return location + 2;
			case 4 : /* slash-slash comment */
				location = text.indexOf(lineSep, separLocation + 2);




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal.consumable;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.STextCharTypes;
import org.eclipse.equinox.bidi.custom.STextProcessor;


		super("+-/*()="); //$NON-NLS-1$
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment
	 *                language) is Arabic.</li>
	 *            <li>The first strong character is an Arabic letter.</li>
	 *            <li>If there is no strong character in the text, there is
	 *                at least one Arabic-Indic digit in the text.</li>
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text) {
		return getDirection(environment, text, new STextCharTypes(text));
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment

	 *            <li>If there is no strong character in the text, there is
	 *                at least one Arabic-Indic digit in the text.</li>
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text, STextCharTypes dirProps) {
		if (environment == null)
			environment = STextEnvironment.DEFAULT;
		String language = environment.getLanguage();
		if (!language.equals("ar")) //$NON-NLS-1$
			return STextHandler.DIR_LTR;
		boolean flagAN = false;
		for (int i = 0; i < text.length(); i++) {
			byte dirProp = dirProps.getBidiTypeAt(i);
			if (dirProp == AL)
				return STextHandler.DIR_RTL;
			if (dirProp == L || dirProp == R)
				return STextHandler.DIR_LTR;
			if (dirProp == AN)
				flagAN = true;
		}
		if (flagAN)
			return STextHandler.DIR_RTL;
		return STextHandler.DIR_LTR;
	}

}




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal.consumable;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.*;

/**
 *  <code>STextRegex</code> is a processor for regular expressions.

 *  <p>
 *  In applications like an editor where parts of the text might be modified
 *  while other parts are not, the user may want to call
 *  {@link STextHandler#leanToFullText leanToFullText}
 *  separately on each line and save the initial <b>state</b> of each line (this is
 *  the final <b>state</b> of the previous line which can be retrieved using
 *  {@link STextHandler#getState}.
 *  If both the content of a line and its initial 
 *  <b>state</b> have not changed, the user can be sure that
 *  the last <i>full</i> text computed for this line has not changed either.
 *
 *  @see STextEngine#leanToFullText explanation of state in leanToFullText
 *
 *  @author Matitiahu Allouche
 */

	 *  This method locates occurrences of the syntactic strings and of
	 *  R, AL, EN, AN characters.
	 */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		// In this method, L, R, AL, AN and EN represent bidi categories
		// as defined in the Unicode Bidirectional Algorithm
		// ( http://www.unicode.org/reports/tr9/ ).

		// AN represents the category Arabic Number.
		// EN  represents the category European Number.
		byte dirProp;
		String text = procData.text;

		if (caseNumber < numberOfStrings) {
			/*  1 *//* comment (?#...) */

		// there never is a need for a mark before the first char
		if (fromIndex <= 0)
			fromIndex = 1;
		STextCharTypes dirProps = procData.dirProps;
		// look for R, AL, AN, EN which are potentially needing a mark
		for (; fromIndex < text.length(); fromIndex++) {
			dirProp = dirProps.getBidiTypeAt(fromIndex);

	/**
	 *  This method process the special cases.
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		int location;
		String text = procData.text;

		switch (caseNumber) {
			case 1 : /* comment (?#...) */

					// initial state from previous line
					location = 0;
				} else {
					STextProcessor.processSeparator(procData, separLocation);
					// skip the opening "(?#"
					location = separLocation + 3;
				}
				location = text.indexOf(')', location);
				if (location < 0) {
					procData.handler.setState(caseNumber);
					return text.length();
				}
				return location + 1;

			case 5 : /* conditional named back reference (?(<name>) */
			case 6 : /* conditional named back reference (?('name') */
			case 7 : /* named parentheses reference (?&name) */
				STextProcessor.processSeparator(procData, separLocation);
				// no need for calling processSeparator() for the following cases
				//   since the starting string contains a L char
			case 8 : /* named group (?P<name> */

					// initial state from previous line
					location = 0;
				} else {
					STextProcessor.processSeparator(procData, separLocation);
					// skip the opening "\Q"
					location = separLocation + 2;
				}
				location = text.indexOf("\\E", location); //$NON-NLS-1$
				if (location < 0) {
					procData.handler.setState(caseNumber);
					return text.length();
				}
				// set the dirProp for the "E" to L (Left to Right character)
				procData.dirProps.setBidiTypeAt(location + 1, L);
				return location + 2;
			case 18 : /* R, AL, AN, EN */
				STextProcessor.processSeparator(procData, separLocation);
				return separLocation + 1;

		}

		return text.length();
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment
	 *                language) is Arabic.</li>
	 *            <li>The first strong character is an Arabic letter.</li>
	 *            <li>If there is no strong character in the text, there is
	 *                at least one Arabic-Indic digit in the text.</li>
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text) {
		return getDirection(environment, text, new STextCharTypes(text));
	}

	/**
	 *  @return {@link STextHandler#DIR_RTL DIR_RTL} if the following
	 *          conditions are satisfied:
	 *          <ul>
	 *            <li>The current locale (as expressed by the environment

	 *            <li>If there is no strong character in the text, the
	 *                GUI is mirrored.
	 *          </ul>
	 *          Otherwise, returns {@link STextHandler#DIR_LTR DIR_LTR}.
	 */
	public int getDirection(STextEnvironment environment, String text, STextCharTypes dirProps) {
		if (environment == null)
			environment = STextEnvironment.DEFAULT;
		String language = environment.getLanguage();
		if (!language.equals("ar")) //$NON-NLS-1$
			return STextHandler.DIR_LTR;
		for (int i = 0; i < text.length(); i++) {
			byte dirProp = dirProps.getBidiTypeAt(i);
			if (dirProp == AL || dirProp == R)
				return STextHandler.DIR_RTL;
			if (dirProp == L)
				return STextHandler.DIR_LTR;
		}
		if (environment.getMirrored())
			return STextHandler.DIR_RTL;
		return STextHandler.DIR_LTR;
	}

}




 ******************************************************************************/
package org.eclipse.equinox.bidi.internal.consumable;

import org.eclipse.equinox.bidi.STextEngine;
import org.eclipse.equinox.bidi.STextEnvironment;
import org.eclipse.equinox.bidi.STextHandler;
import org.eclipse.equinox.bidi.custom.STextProcessor;
import org.eclipse.equinox.bidi.custom.STextProcessorData;
import org.eclipse.equinox.bidi.internal.STextActivator;

/**

 *  <p>
 *  In applications like an editor where parts of the text might be modified
 *  while other parts are not, the user may want to call
 *  {@link STextHandler#leanToFullText leanToFullText}
 *  separately on each line and save the initial <b>state</b> of each line (this is
 *  the final <b>state</b> of the previous line which can be retrieved using
 *  {@link STextHandler#getState}.
 *  If both the content of a line and its initial
 *  <b>state</b> have not changed, the user can be sure that
 *  the last <i>full</i> text computed for this line has not changed either.
 *
 *  @see STextEngine#leanToFullText explanation of state in leanToFullText
 *
 *  @author Matitiahu Allouche
 */

	  *    <li>comments starting with hyphen-hyphen</li>
	  *  </ol>
	  */
	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
		String text = procData.text;
		switch (caseNumber) {
			case 1 : /* space */
				return text.indexOf(" ", fromIndex); //$NON-NLS-1$

	     *    <li>skip until after a line separator</li>
	     *  </ol>
	 */
	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
		int location;
		String text = procData.text;

		STextProcessor.processSeparator(procData, separLocation);
		switch (caseNumber) {
			case 1 : /* space */
				separLocation++;
				while (separLocation < text.length() && text.charAt(separLocation) == ' ') {
					procData.dirProps.setBidiTypeAt(separLocation, WS);
					separLocation++;
				}
				return separLocation;

				while (true) {
					location = text.indexOf('\'', location);
					if (location < 0) {
						procData.handler.setState(caseNumber);
						return text.length();
					}
					if ((location + 1) < text.length() && text.charAt(location + 1) == '\'') {

					location = separLocation + 2; // skip the opening slash-aster
				location = text.indexOf("*/", location); //$NON-NLS-1$
				if (location < 0) {
					procData.handler.setState(caseNumber);
					return text.length();
				}
				// we need to call processSeparator since text may follow the
				//  end of comment immediately without even a space
				STextProcessor.processSeparator(procData, location);
				return location + 2;
			case 5 : /* hyphen-hyphen comment */
				location = text.indexOf(lineSep, separLocation + 2);




<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<META name="Author" content="Matitiahu Allouche">
</head>
<body bgcolor="white">

This package provides classes for
processing structured text.
<p>
There are various types of structured text. Each type should
be handled by a specific <i>processor</i>. A number of standard
processors are supplied in the associated package
{@link <a href="internal\consumable\package-summary.html">
org.eclipse.equinox.bidi.internal.consumable</a>}.

<h2>Introduction to Structured Text</h2>
<p>
Bidirectional text offers interesting challenges to presentation systems.
For plain text, the Unicode Bidirectional Algorithm
(<a href="http://www.unicode.org/reports/tr9/">UBA</a>)
generally specifies satisfactorily how to reorder bidirectional text for
display. This algorithm is implemented in Java's presentation system.
</p><p>
However, all bidirectional text is not necessarily plain text. There
are also instances of text structured to follow a given syntax, which
should be reflected in the display order. The general algorithm, which
has no awareness of these special cases, often gives incorrect results
when displaying such structured text.
</p><p>
The general idea in handling structured text is to add directional
formatting characters at proper locations in the text to supplement the
standard algorithm, so that the final result is correctly displayed
using the UBA.
</p><p>
A class which handles structured text is thus essentially a
transformation engine which receives text without directional formatting
characters as input and produces as output the same text with added
directional formatting characters, hopefully in the minimum quantity
which is sufficient to ensure correct display, considering the type of
structured text involved.
</p><p>
In this package, text without directional formatting characters is
called <b><i>lean</i></b> text while the text with added directional
formatting characters is called <b><i>full</i></b> text.
</p><p>
The class {@link
<a href="STextHandler.html"><b>STextHandler</b></a>}
is the main tool for processing structured text.  It facilitates
handling several types of structured text, each type being handled
by a specific
{@link <a href="custom\STextProcessor.html"><b><i>processor</i></b></a>} :</p>
<ul>
  <li>property (name=value)</li>
  <li>compound name (xxx_yy_zzzz)</li>
  <li>comma delimited list</li>
  <li>system(user)</li>
  <li>directory and file path</li>
  <li>e-mail address</li>
  <li>URL</li>
  <li>regular expression</li>
  <li>Xpath</li>
  <li>Java code</li>
  <li>SQL statements</li>
  <li>RTL arithmetic expressions</li>
</ul>
<p>
For each of these types, a static instance is defined in
{@link <a href="STextProcessorFactory.html">STextProcessorFactory</a>}.
These pre-defined instances can be used as argument when initiating a
<b>STextHandler</b>.
</p><p>
Other classes in this package may be used to
complement and facilitate the action of
{@link <a href="STextHandler.html">STextHandler</a>}:
<ul>
  <li>{@link <a href="STextEnvironment.html">STextEnvironment</a>}
      regroups details about the environment.</li>
  <li>{@link <a href="STextProcessorFactory.html">STextProcessorFactory</a>}
      allows retrieval of the defined processor types and of the
      corresponding processors.</li>
  <li>{@link <a href="STextUtil.html">STextUtil</a>}
      provides a number of convenience methods to process some common types of
      structured text.  When using methods in this class, there is no need
      to use other classes of this package.  However, the other classes allow
      more precise control and possibly better performance.</li>
  <li>{@link <a href="STextStringRecord.html">STextStringRecord</a>}
      allows to record strings which are structured text with the
      type of the appropriate processor,
      so that another module can check if a given string has been recorded
      as a structured text string and run its processor.</li>
</ul>
<p>
{@link <a href="STextHandler.html">STextHandler</a>} and the
other classes mentioned above are intended for users who
need to process structured text for which there already exist
processors.
<p>
Developers who want to develop new processors to support types of structured text
not currently supported can use the following components of the
package {@link <a href="custom\package-summary.html">
org.eclipse.equinox.bidi.custom</a>}:
<ul>
  <li>{@link <a href="custom\STextProcessor.html">STextProcessor</a>}
      is a generic processor to be used as superclass for specific
      processors.</li>
  <li>{@link <a href="custom\STextProcessorData.html">STextProcessorData</a>}
      regroups data which can be used by processors.</li>
  <li>{@link <a href="custom\STextCharTypes.html">STextStringCharTypes</a>}
      allows getting and setting the bidi class of characters.</li>
  <li>{@link <a href="custom\STextOffsets.html">STextStringOffsets</a>}
      allows manipulation of offsets at which directional control
      characters are inserted.</li>
</ul>
<p>
There are two other packages associated with the current one:
<ul>
  <li>{@link <a href="internal\package-summary.html">
      org.eclipse.equinox.bidi.internal</a>}</li>
  <li>{@link <a href="internal\consumable\package-summary.html">
       org.eclipse.equinox.bidi.internal.consumable</a>}</li>
</ul>
The tools in the first package can serve as example of how to develop
processors for currently unsupported types of structured text.<br>
The latter package contains classes for the processors implementing
the currently supported types of structured text.
<p>
However, users wishing to process the currently supported types of
structured text typically don't need to interact with these
two packages.

<p>&nbsp;</p>

<h2>Abbreviations used in the documentation of this package</h2>

<dl>
<dt><b>UBA</b><dd>Unicode Bidirectional Algorithm

<dt><b>Bidi</b><dd>Bidirectional

<dt><b>GUI</b><dd>Graphical User Interface

<dt><b>UI</b><dd>User Interface

<dt><b>LTR</b><dd>Left to Right

<dt><b>RTL</b><dd>Right to Left

<dt><b>LRM</b><dd>Left-to-Right Mark

<dt><b>RLM</b><dd>Right-to-Left Mark

<dt><b>LRE</b><dd>Left-to-Right Embedding

<dt><b>RLE</b><dd>Right-to-Left Embedding

<dt><b>PDF</b><dd>Pop Directional Formatting
</dl>

<p>&nbsp;</p>

<h2>Known Limitations</h2>

<p>The proposed solution is making extensive usage of LRM, RLM, LRE, RLE
and PDF directional controls which are invisible but affect the way bidi
text is displayed. The following related key points merit special
attention:</p>

<ul>
<li>Implementations of the UBA on various platforms (e.g., Windows and
Linux) are very similar but nevertheless have known differences. Those
differences are minor and will not have a visible effect in most cases.
However there might be cases in which the same bidi text on two
platforms might look different. These differences will surface in Java
applications when they use the platform visual components for their UI
(e.g., AWT, SWT).</li>

<li>It is assumed that the presentation engine supports LRE, RLE and
PDF directional formatting characters.</li>

<li>Because some presentation engines are not strictly conformant to the
UBA, the implementation of structured text in this package adds LRM
or RLM characters in association with LRE, RLE or PDF in cases where
this would not be needed if the presentation engine was fully conformant
to the UBA. Such added marks will not have harmful effects on
conformant presentation engines and will help less conformant engines to
achieve the desired presentation.</li>
</ul>

</body>
</html>

Return to bug 183164

Lines 1-318 Link Here

(-)src/org/eclipse/equinox/bidi/STextEngine.java (-318 lines)
1	/*******************************************************************************
2	* Copyright (c) 2010, 2011 IBM Corporation and others.
3	* All rights reserved. This program and the accompanying materials
4	* are made available under the terms of the Eclipse Public License v1.0
5	* which accompanies this distribution, and is available at
6	* http://www.eclipse.org/legal/epl-v10.html
7	*
8	* Contributors:
9	* IBM Corporation - initial API and implementation
10	******************************************************************************/
11	package org.eclipse.equinox.bidi;
12
13	import org.eclipse.equinox.bidi.custom.STextProcessor;
14	import org.eclipse.equinox.bidi.internal.STextImpl;
15
16	/**
17	* For a general introduction to structured text, see
18	* {@link <a href="package-summary.html"> the package documentation</a>}.
19	* <p>
20	* Several common processors are included in <b>STextEngine</b>. For processors
21	* supplied by other packages, a processor instance can be obtained using the
22	* {@link org.eclipse.equinox.bidi.STextProcessorFactory#getProcessor}
23	* method for the registered processors, or by instantiating a private processor.
24	* </p><p>
25	* Most of the methods in this class have a <code>text</code>
26	* argument which may be just a part of a larger body of text.
27	* When it is the case that the text is submitted in parts with
28	* repeated calls, there may be a need to pass information from
29	* one invocation to the next one. For instance, one invocation
30	* may detect that a comment or a literal has been started but
31	* has not been completed. In such cases, a <code>state</code>
32	* argument must be used.
33	* </p><p>
34	* The <code>state</code> argument must be an array of integers
35	* with at least one element. Only the first element is used by
36	* the methods of this class.
37	* </p><p>
38	* When submitting the initial part of the text, the first element
39	* of <code>state</code> must contain the value {@link #STATE_INITIAL}
40	* or any value <= 0.
41	* </p><p>
42	* After calling a method with a non-null <code>state</code> argument,
43	* a value is returned in the first element of <code>state</code>. This
44	* value should be passed unmodified to the method when calling it again
45	* with the text which is the continuation of the text submitted in the
46	* last call.
47	* </p><p>
48	* When the text submitted to a method is not a continuation and is not
49	* expected to have a continuation , e.g. it is processed all by itself,
50	* the <code>state</code> argument should be specified as <code>null</code>.
51	* </p><p>
52	* <b>Code Samples</b>
53	* </p><p>
54	* The following code shows how to transform a certain type of structured text
55	* (directory and file paths) in order to obtain the <i>full</i>
56	* text corresponding to the given <i>lean</i> text.
57	* <pre>
58	* String leanText = "D:\\\u05d0\u05d1\\\u05d2\\\u05d3.ext";
59	* String fullText = STextEngine.leanToFullText(STextEngine.PROC_FILE, null, leanText, null);
60	* System.out.println("full text = " + fullText);
61	* </pre>
62	* </p><p>
63	* The following code shows how to transform successive lines of Java
64	* code in order to obtain the <i>full</i>
65	* text corresponding to the <i>lean</i> text of each line.
66	* <pre>
67	* int[] state = new int[1];
68	* state[0] = STextEngine.STATE_INITIAL;
69	* String leanText = "int i = 3; // first Java statement";
70	* String fullText = STextEngine.leanToFullText(STextEngine.PROC_JAVA, null, leanText, state);
71	* System.out.println("full text = " + fullText);
72	* leanText = "i += 4; // next Java statement";
73	* fullText = STextEngine.leanToFullText(STextEngine.PROC_JAVA, null, leanText, state);
74	* System.out.println("full text = " + fullText);
75	* </pre>
76	* </p>
77	* @author Matitiahu Allouche
78	*
79	*/
80	public class STextEngine {
81
82	/**
83	* Constant specifying that the base direction of a structured text is LTR.
84	* The base direction may depend on whether the GUI is
85	* {@link STextEnvironment#getMirrored mirrored} and may
86	* may be different for Arabic and for Hebrew.
87	* This constant can appear as value returned by the
88	* {@link #getCurDirection getCurDirection} method.
89	*/
90	public static final int DIR_LTR = 0;
91
92	/**
93	* Constant specifying that the base direction of a structured text is RTL.
94	* The base direction may depend on whether the GUI is
95	* {@link STextEnvironment#getMirrored mirrored} and may
96	* may be different for Arabic and for Hebrew.
97	* This constant can appear as value returned by the
98	* {@link #getCurDirection getCurDirection} method.
99	*/
100	public static final int DIR_RTL = 1;
101
102	/**
103	* Constant to use in the first element of the <code>state</code>
104	* argument when calling most methods of this class
105	* to indicate that there is no context of previous lines which
106	* should be initialized before performing the operation.
107	*/
108	public static final int STATE_INITIAL = 0;
109
110	private static final int[] EMPTY_INT_ARRAY = new int[0];
111
112	/**
113	* Prevent creation of a STextEngine instance
114	*/
115	private STextEngine() {
116	// nothing to do
117	}
118
119	/**
120	* Add directional formatting characters to a structured text
121	* to ensure correct presentation.
122	*
123	* @param processor the processor applicable to the text. If <code>null</code>,
124	* the method returns unmodified text.
125	*
126	* @param environment a bidi environment. If <code>null</code>, the default environment
127	* is used.
128	*
129	* @param text is the structured text string
130	*
131	* @param state can be used to specify that the <code>text</code> argument is
132	* the continuation of text submitted in a previous call and/or to receive information
133	* to pass to continuation calls. If all calls to this method are independent from one another,
134	* this argument should be specified as <code>null</code>.
135	*
136	* @return the structured text with directional formatting characters added to ensure
137	* correct presentation.
138	*/
139	public static String leanToFullText(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
140	if (processor == null)
141	return text;
142	return STextImpl.leanToFullText(processor, environment, text, state);
143	}
144
145	/**
146	* Given a <i>lean</i> string, compute the positions of each of its
147	* characters within the corresponding <i>full</i> string.
148	*
149	* @param processor designates a processor instance. If <code>null</code>, this
150	* method returns an identity map.
151	*
152	* @param environment specifies an environment whose characteristics may affect
153	* the processor's behavior. If <code>null</code>, the default environment is used.
154	*
155	* @param text is the structured text string.
156	*
157	* @param state can be used to specify that the <code>text</code> argument is
158	* the continuation of text submitted in a previous call and/or to receive information
159	* to pass to continuation calls. If all calls to this method are independent from one another,
160	* this argument should be specified as <code>null</code>.
161	*
162	* @return an array which specifies offsets of the <code>text</code> characters
163	* in the <i>full</i> string
164	*/
165	public static int[] leanToFullMap(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
166	if (processor == null) {
167	int[] map = new int[text.length()];
168	for (int i = 0; i < map.length; i++)
169	map[i] = i;
170	return map;
171	}
172	return STextImpl.leanToFullMap(processor, environment, text, state);
173	}
174
175	/**
176	* Given a <i>lean</i> string, compute the offsets of characters
177	* before which directional formatting characters must be added
178	* in order to ensure correct presentation.
179	* <p>
180	* Only LRMs (for a string with LTR base direction) and RLMs (for
181	* a string with RTL base direction) are considered. Leading and
182	* trailing LRE, RLE and PDF which might be prefixed or suffixed
183	* depending on the {@link STextEnvironment#getOrientation orientation} of the
184	* GUI component used for display are not reflected in this method.
185	* </p>
186	* @param processor designates a processor instance
187	*
188	* @param environment specifies an environment whose characteristics may affect
189	* the processor's behavior. If <code>null</code>, the default environment is used.
190	*
191	* @param text is the structured text string
192	*
193	* @param state can be used to specify that the <code>text</code> argument is
194	* the continuation of text submitted in a previous call and/or to receive information
195	* to pass to continuation calls. If all calls to this method are independent from one another,
196	* this argument should be specified as <code>null</code>.
197	*
198	* @return an array of offsets to the characters in the <code>text</code> argument
199	* before which directional marks must be added to ensure correct presentation.
200	* The offsets are sorted in ascending order.
201	*/
202	public static int[] leanBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
203	if (processor == null)
204	return EMPTY_INT_ARRAY;
205	return STextImpl.leanBidiCharOffsets(processor, environment, text, state);
206	}
207
208	/**
209	* Remove directional formatting characters which were added to a
210	* structured text string to ensure correct presentation.
211	*
212	* @param processor designates a processor instance
213	*
214	* @param environment specifies an environment whose characteristics may affect
215	* the processor's behavior. If <code>null</code>, the default environment is used.
216	*
217	* @param text is the structured text string including directional formatting characters.
218	*
219	* @param state can be used to specify that the <code>text</code> argument is
220	* the continuation of text submitted in a previous call and/or to receive information
221	* to pass to continuation calls. If all calls to this method are independent from one another,
222	* this argument should be specified as <code>null</code>.
223	*
224	* @return the structured text string without directional formatting characters
225	* which might have been added by processing it with {@link #leanToFullText}.
226	*
227	*/
228	public static String fullToLeanText(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
229	if (processor == null)
230	return text;
231	return STextImpl.fullToLeanText(processor, environment, text, state);
232	}
233
234	/**
235	* Given a <i>full</i> string, compute the positions of each of its
236	* characters within the corresponding <i>lean</i> string.
237	*
238	* @param processor designates a processor instance
239	*
240	* @param environment specifies an environment whose characteristics may affect
241	* the processor's behavior. If <code>null</code>, the default environment is used.
242	*
243	* @param text is the structured text string including directional formatting characters.
244	*
245	* @param state can be used to specify that the <code>text</code> argument is
246	* the continuation of text submitted in a previous call and/or to receive information
247	* to pass to continuation calls. If all calls to this method are independent from one another,
248	* this argument should be specified as <code>null</code>.
249	*
250	* @return an array of integers with one element for each of the characters
251	* in the <code>text</code> argument, equal to the offset of the corresponding character
252	* in the <i>lean</i> string. If there is no corresponding character in the <i>lean</i> string
253	* (because the specified character is a directional formatting character added when invoking
254	* {@link #leanToFullText}), the value returned for this character is -1.
255	*/
256	public static int[] fullToLeanMap(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
257	if (processor == null) {
258	int[] map = new int[text.length()];
259	for (int i = 0; i < map.length; i++)
260	map[i] = i;
261	return map;
262	}
263	return STextImpl.fullToLeanMap(processor, environment, text, state);
264	}
265
266	/**
267	* Given a <i>full</i> string, return the offsets of characters
268	* which are directional formatting characters that have been added
269	* in order to ensure correct presentation.
270	* <p>
271	* LRMs (for a string with LTR base direction), RLMs (for a string with RTL base direction)
272	* are considered as well as leading and trailing LRE, RLE and PDF which might be prefixed
273	* or suffixed depending on the {@link STextEnvironment#getOrientation orientation}
274	* of the GUI component used for display.
275	* </p>
276	* @param processor designates a processor instance
277	*
278	* @param environment specifies an environment whose characteristics may affect
279	* the processor's behavior. If <code>null</code>, the default environment is used.
280	*
281	* @param text is the structured text string including directional formatting characters
282	*
283	* @param state can be used to specify that the <code>text</code> argument is
284	* the continuation of text submitted in a previous call and/or to receive information
285	* to pass to continuation calls. If all calls to this method are independent from one another,
286	* this argument should be specified as <code>null</code>.
287	*
288	* @return an array of offsets to the characters in the <code>text</code> argument which
289	* are directional formatting characters added to ensure correct presentation. The offsets
290	* are sorted in ascending order.
291	*/
292	public static int[] fullBidiCharOffsets(STextProcessor processor, STextEnvironment environment, String text, int[] state) {
293	if (processor == null)
294	return EMPTY_INT_ARRAY;
295	return STextImpl.fullBidiCharOffsets(processor, environment, text, state);
296	}
297
298	/**
299	* Get the base direction of a structured text. This base direction may depend on
300	* whether the text contains Arabic or Hebrew words. If the text contains both,
301	* the first Arabic or Hebrew letter in the text determines which is the governing script.
302	*
303	* @param processor designates a processor instance
304	*
305	* @param environment specifies an environment whose characteristics may affect
306	* the processor's behavior. If <code>null</code>, the default environment is used.
307	*
308	* @param text is the structured text string
309	*
310	* @return the base direction of the structured text, {@link #DIR_LTR} or {@link #DIR_RTL}
311	*/
312	public static int getCurDirection(STextProcessor processor, STextEnvironment environment, String text) {
313	if (processor == null)
314	return DIR_LTR;
315	return processor.getDirection(environment, text);
316	}
317
318	}

Added Link Here

(-)src/org/eclipse/equinox/bidi/STextHandler.java (+316 lines)
1	/*******************************************************************************
2	* Copyright (c) 2010, 2011 IBM Corporation and others.
3	* All rights reserved. This program and the accompanying materials
4	* are made available under the terms of the Eclipse Public License v1.0
5	* which accompanies this distribution, and is available at
6	* http://www.eclipse.org/legal/epl-v10.html
7	*
8	* Contributors:
9	* IBM Corporation - initial API and implementation
10	******************************************************************************/
11	package org.eclipse.equinox.bidi;
12
13	import org.eclipse.equinox.bidi.custom.*;
14	import org.eclipse.equinox.bidi.internal.STextImpl;
15
16	/**
17	* For a general introduction to structured text, see
18	* {@link <a href="package-summary.html"> the package documentation</a>}.
19	* <p>
20	* Several common processors are included in
21	* {@link org.eclipse.equinox.bidi.STextProcessorFactory}.
22	* For processors
23	* supplied by other packages, a processor instance can be obtained using the
24	* {@link org.eclipse.equinox.bidi.STextProcessorFactory#getProcessor}
25	* method for the registered processors, or by instantiating a private processor.
26	* </p><p>
27	* Most of the methods in this class have a <code>text</code>
28	* argument which may be just a part of a larger body of text.
29	* When it is the case that the text is submitted in parts with
30	* repeated calls, there may be a need to pass information from
31	* one invocation to the next one. For instance, one invocation
32	* may detect that a comment or a literal has been started but
33	* has not been completed.
34	* </p><p>
35	* This information is summarized as one integer, the "<b>state</b>".
36	* The <b>state</b> can be queried using {@link #getState} after any
37	* operation, and it can be set before an operation using
38	* {@link #setState}.
39	* </p><p>
40	* When submitting the initial part of the text, the <b>state</b>
41	* must contain the value {@link #STATE_INITIAL} or any value <= 0.
42	* </p><p>
43	* When calling a method repeatedly, the <b>state</b> is automatically
44	* passed from one invocation to the next unless it is reset
45	* by the caller using {@link #setState}.
46	* </p><p>
47	* <b>Code Samples</b>
48	* </p><p>
49	* The following code shows how to transform a certain type of structured text
50	* (directory and file paths) in order to obtain the <i>full</i>
51	* text corresponding to the given <i>lean</i> text.
52	* <pre>
53	* STextHandler handler = new STextHandler(STextProcessorFactory.PROC_FILE);
54	* String leanText = "D:\\\u05d0\u05d1\\\u05d2\\\u05d3.ext";
55	* String fullText = handler.leanToFullText(null, leanText);
56	* System.out.println("full text = " + fullText);
57	* </pre>
58	* </p><p>
59	* The following code shows how to transform successive lines of Java
60	* code in order to obtain the <i>full</i>
61	* text corresponding to the <i>lean</i> text of each line.
62	* <pre>
63	* STextHandler handler = new STextHandler(STextProcessorFactory.PROC_JAVA);
64	* String leanText = "int i = 3; // first Java statement";
65	* String fullText = handler.leanToFullText(null, leanText);
66	* System.out.println("full text = " + fullText);
67	* leanText = "i += 4; // next Java statement";
68	* fullText = handler.leanToFullText(null, leanText);
69	* System.out.println("full text = " + fullText);
70	* </pre>
71	* </p>
72	* @author Matitiahu Allouche
73	*
74	*/
75	public class STextHandler {
76
77	/**
78	* Constant specifying that the base direction of a structured text is LTR.
79	* The base direction may depend on whether the GUI is
80	* {@link STextEnvironment#getMirrored mirrored} and may
81	* may be different for Arabic and for Hebrew.
82	* This constant can appear as value returned by the
83	* {@link #getCurDirection getCurDirection} method.
84	*/
85	public static final int DIR_LTR = 0;
86
87	/**
88	* Constant specifying that the base direction of a structured text is RTL.
89	* The base direction may depend on whether the GUI is
90	* {@link STextEnvironment#getMirrored mirrored} and may
91	* may be different for Arabic and for Hebrew.
92	* This constant can appear as value returned by the
93	* {@link #getCurDirection getCurDirection} method.
94	*/
95	public static final int DIR_RTL = 1;
96
97	/**
98	* Constant to use in {@link #setState}
99	* to indicate that there is no context of previous lines
100	* to consider before performing the next operation.
101	*/
102	public static final int STATE_INITIAL = 0;
103
104	private static final int[] EMPTY_INT_ARRAY = new int[0];
105
106	private STextProcessorData procData;
107	private int state;
108
109	/**
110	* Constructor
111	*
112	* @param processor is the processor which will transform the
113	* submitted text.
114	*/
115	public STextHandler(STextProcessor processor) {
116	procData = new STextProcessorData();
117	setProcessor(processor);
118	procData.handler = this;
119	procData.offsets = new STextOffsets();
120	}
121
122	/**
123	* Replace the processor of an existing handler. This operation
124	* resets the state to {@link #STATE_INITIAL}.
125	*
126	* @param processor is the new processor.
127	*/
128	public void setProcessor(STextProcessor processor) {
129	procData.processor = processor;
130	state = STATE_INITIAL;
131	}
132
133	/**
134	* Set the state for the next operation.
135	*
136	* @param state is the new value.
137	*/
138	public void setState(int state) {
139	this.state = state;
140	}
141
142	/**
143	* Get the value of the state after the last operation.
144	*
145	* @return the last value of the state.
146	*/
147	public int getState() {
148	return state;
149	}
150
151	/**
152	* Add directional formatting characters to a structured text
153	* to ensure correct presentation.
154	*
155	* @param environment a bidi environment. If <code>null</code>, the default environment
156	* is used.
157	*
158	* @param text is the structured text string
159	*
160	* @return the structured text with directional formatting characters added to ensure
161	* correct presentation.
162	*/
163	public String leanToFullText(STextEnvironment environment, String text) {
164	if (procData.processor == null)
165	return text;
166	procData.environment = environment;
167	procData.text = text;
168	return STextImpl.leanToFullText(procData);
169	}
170
171	/**
172	* Given a <i>lean</i> string, compute the positions of each of its
173	* characters within the corresponding <i>full</i> string.
174	*
175	* @param environment specifies an environment whose characteristics may affect
176	* the processor's behavior. If <code>null</code>, the default environment is used.
177	*
178	* @param text is the structured text string.
179	*
180	* @return an array which specifies offsets of the <code>text</code> characters
181	* in the <i>full</i> string
182	*/
183	public int[] leanToFullMap(STextEnvironment environment, String text) {
184	if (procData.processor == null) {
185	int[] map = new int[text.length()];
186	for (int i = 0; i < map.length; i++)
187	map[i] = i;
188	return map;
189	}
190	procData.environment = environment;
191	procData.text = text;
192	return STextImpl.leanToFullMap(procData);
193	}
194
195	/**
196	* Given a <i>lean</i> string, compute the offsets of characters
197	* before which directional formatting characters must be added
198	* in order to ensure correct presentation.
199	* <p>
200	* Only LRMs (for a string with LTR base direction) and RLMs (for
201	* a string with RTL base direction) are considered. Leading and
202	* trailing LRE, RLE and PDF which might be prefixed or suffixed
203	* depending on the {@link STextEnvironment#getOrientation orientation} of the
204	* GUI component used for display are not reflected in this method.
205	* </p>
206	* @param environment specifies an environment whose characteristics may affect
207	* the processor's behavior. If <code>null</code>, the default environment is used.
208	*
209	* @param text is the structured text string
210	*
211	* @return an array of offsets to the characters in the <code>text</code> argument
212	* before which directional marks must be added to ensure correct presentation.
213	* The offsets are sorted in ascending order.
214	*/
215	public int[] leanBidiCharOffsets(STextEnvironment environment, String text) {
216	if (procData.processor == null)
217	return EMPTY_INT_ARRAY;
218	procData.environment = environment;
219	procData.text = text;
220	return STextImpl.leanBidiCharOffsets(procData);
221	}
222
223	/**
224	* Remove directional formatting characters which were added to a
225	* structured text string to ensure correct presentation.
226	*
227	* @param environment specifies an environment whose characteristics may affect
228	* the processor's behavior. If <code>null</code>, the default environment is used.
229	*
230	* @param text is the structured text string including directional formatting characters.
231	*
232	* @return the structured text string without directional formatting characters
233	* which might have been added by processing it with {@link #leanToFullText}.
234	*
235	*/
236	public String fullToLeanText(STextEnvironment environment, String text) {
237	if (procData.processor == null)
238	return text;
239	procData.environment = environment;
240	procData.text = text;
241	return STextImpl.fullToLeanText(procData);
242	}
243
244	/**
245	* Given a <i>full</i> string, compute the positions of each of its
246	* characters within the corresponding <i>lean</i> string.
247	*
248	* @param environment specifies an environment whose characteristics may affect
249	* the processor's behavior. If <code>null</code>, the default environment is used.
250	*
251	* @param text is the structured text string including directional formatting characters.
252	*
253	* @return an array of integers with one element for each of the characters
254	* in the <code>text</code> argument, equal to the offset of the corresponding character
255	* in the <i>lean</i> string. If there is no corresponding character in the <i>lean</i> string
256	* (because the specified character is a directional formatting character added when invoking
257	* {@link #leanToFullText}), the value returned for this character is -1.
258	*/
259	public int[] fullToLeanMap(STextEnvironment environment, String text) {
260	if (procData.processor == null) {
261	int[] map = new int[text.length()];
262	for (int i = 0; i < map.length; i++)
263	map[i] = i;
264	return map;
265	}
266	procData.environment = environment;
267	procData.text = text;
268	return STextImpl.fullToLeanMap(procData);
269	}
270
271	/**
272	* Given a <i>full</i> string, return the offsets of characters
273	* which are directional formatting characters that have been added
274	* in order to ensure correct presentation.
275	* <p>
276	* LRMs (for a string with LTR base direction), RLMs (for a string with RTL base direction)
277	* are considered as well as leading and trailing LRE, RLE and PDF which might be prefixed
278	* or suffixed depending on the {@link STextEnvironment#getOrientation orientation}
279	* of the GUI component used for display.
280	* </p>
281	* @param environment specifies an environment whose characteristics may affect
282	* the processor's behavior. If <code>null</code>, the default environment is used.
283	*
284	* @param text is the structured text string including directional formatting characters
285	*
286	* @return an array of offsets to the characters in the <code>text</code> argument which
287	* are directional formatting characters added to ensure correct presentation. The offsets
288	* are sorted in ascending order.
289	*/
290	public int[] fullBidiCharOffsets(STextEnvironment environment, String text) {
291	if (procData.processor == null)
292	return EMPTY_INT_ARRAY;
293	procData.environment = environment;
294	procData.text = text;
295	return STextImpl.fullBidiCharOffsets(procData);
296	}
297
298	/**
299	* Get the base direction of a structured text. This base direction may depend on
300	* whether the text contains Arabic or Hebrew words. If the text contains both,
301	* the first Arabic or Hebrew letter in the text determines which is the governing script.
302	*
303	* @param environment specifies an environment whose characteristics may affect
304	* the processor's behavior. If <code>null</code>, the default environment is used.
305	*
306	* @param text is the structured text string
307	*
308	* @return the base direction of the structured text, {@link #DIR_LTR} or {@link #DIR_RTL}
309	*/
310	public int getCurDirection(STextEnvironment environment, String text) {
311	if (procData.processor == null)
312	return DIR_LTR;
313	return procData.processor.getDirection(environment, text);
314	}
315
316	}

Lines 16-24 Link Here

(-)src/org/eclipse/equinox/bidi/STextUtil.java (-12 / +10 lines)
16	* This class provides a number of convenience functions facilitating the	16	* This class provides a number of convenience functions facilitating the
17	* processing of structured text.	17	* processing of structured text.
18	*	18	*
19	* @noextend This class is not intended to be subclassed by clients.
20	* @noinstantiate This class is not intended to be instantiated by clients.
21	*
22	* @author Matitiahu Allouche	19	* @author Matitiahu Allouche
23	*/	20	*/
24	public final class STextUtil {	21	public final class STextUtil {
Lines 61-67 Link Here
61	* If necessary, leading and trailing directional markers (LRE, RLE and PDF) can	58	* If necessary, leading and trailing directional markers (LRE, RLE and PDF) can
62	* be added depending on the value of the <code>affix</code> argument.	59	* be added depending on the value of the <code>affix</code> argument.
63	* </p>	60	* </p>
64	* @see STextEngine#leanBidiCharOffsets(STextProcessor, STextEnvironment, String, int[])	61	* @see STextHandler#leanBidiCharOffsets leanBidiCharOffsets
65	*	62	*
66	* @param text the structured text string	63	* @param text the structured text string
67	* @param offsets an array of offsets to characters in <code>text</code>	64	* @param offsets an array of offsets to characters in <code>text</code>
Lines 69-76 Link Here
69	* The array must be sorted in ascending order without duplicates.	66	* The array must be sorted in ascending order without duplicates.
70	* This argument may be <code>null</code> if there are no marks to add.	67	* This argument may be <code>null</code> if there are no marks to add.
71	* @param direction the base direction of the structured text.	68	* @param direction the base direction of the structured text.
72	* It must be one of the values {@link STextEngine#DIR_LTR}, or	69	* It must be one of the values {@link STextHandler#DIR_LTR}, or
73	* {@link STextEngine#DIR_RTL}.	70	* {@link STextHandler#DIR_RTL}.
74	* @param affix specifies if a prefix and a suffix should be added to	71	* @param affix specifies if a prefix and a suffix should be added to
75	* the result	72	* the result
76	* @return a string corresponding to the source <code>text</code> with	73	* @return a string corresponding to the source <code>text</code> with
Lines 86-92 Link Here
86	String curPrefix, curSuffix, full;	83	String curPrefix, curSuffix, full;
87	char curMark, c;	84	char curMark, c;
88	char[] fullChars;	85	char[] fullChars;
89	if (direction == STextEngine.DIR_LTR) {	86	if (direction == STextHandler.DIR_LTR) {
90	curMark = LRM;	87	curMark = LRM;
91	curPrefix = "\u202a\u200e"; /* LRE+LRM *///$NON-NLS-1$	88	curPrefix = "\u202a\u200e"; /* LRE+LRM *///$NON-NLS-1$
92	curSuffix = "\u200e\u202c"; /* LRM+PDF *///$NON-NLS-1$	89	curSuffix = "\u200e\u202c"; /* LRM+PDF *///$NON-NLS-1$
Lines 187-194 Link Here
187	separators = defaultSeparators;	184	separators = defaultSeparators;
188		185
189	// make sure that LRE/PDF are added around the string	186	// make sure that LRE/PDF are added around the string
190	STextProcessor processor = new STextProcessor(separators);	187	STextHandler handler = new STextHandler(new STextProcessor(separators));
191	return STextEngine.leanToFullText(processor, env, str, null);	188	return handler.leanToFullText(env, str);
192	}	189	}
193		190
194	/**	191	/**
Lines 212-218 Link Here
212	STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);	209	STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);
213	if (!env.isProcessingNeeded())	210	if (!env.isProcessingNeeded())
214	return str;	211	return str;
215	return STextEngine.leanToFullText(processor, env, str, null);	212	STextHandler handler = new STextHandler(processor);
		213	return handler.leanToFullText(env, str);
216	}	214	}
217		215
218	/**	216	/**
Lines 259-265 Link Here
259	STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);	257	STextEnvironment env = new STextEnvironment(null, false, STextEnvironment.ORIENT_UNKNOWN);
260	if (!env.isProcessingNeeded())	258	if (!env.isProcessingNeeded())
261	return str;	259	return str;
262	return STextEngine.fullToLeanText(processor, env, str, null);	260	STextHandler handler = new STextHandler(processor);
		261	return handler.fullToLeanText(env, str);
263	}	262	}
264
265	}	263	}

Added Link Here

(-)src/org/eclipse/equinox/bidi/custom/STextOffsets.java (+109 lines)
1	package org.eclipse.equinox.bidi.custom;
2
3	public class STextOffsets {
4	static final byte L = Character.DIRECTIONALITY_LEFT_TO_RIGHT;
5	static final byte R = Character.DIRECTIONALITY_RIGHT_TO_LEFT;
6	static final byte AL = Character.DIRECTIONALITY_RIGHT_TO_LEFT_ARABIC;
7	static final byte AN = Character.DIRECTIONALITY_ARABIC_NUMBER;
8	static final byte EN = Character.DIRECTIONALITY_EUROPEAN_NUMBER;
9	static final byte[] STRONGS = {L, R};
10	static final int OFFSET_SIZE = 20;
11	int[] offsets = new int[OFFSET_SIZE];
12	int count; // number of used entries
13
14	/**
15	* @return the number of used entries in the offsets array.
16	*/
17	public int getCount() {
18	return count;
19	}
20
21	/**
22	* Mark that all entries in the offsets array are unused.
23	*/
24	public void resetCount() {
25	count = 0;
26	}
27
28	/**
29	* Get the value of a specified entry in the offsets array.
30	*
31	* @param index is the index of the entry of interest.
32	*
33	* @return the value of the specified entry.
34	*/
35	public int getOffset(int index) {
36	return offsets[index];
37	}
38
39	/**
40	* Insert an offset value in the offset array so that the array
41	* stays in ascending order.
42	*
43	* @param procData is a group of data accessible to processors.
44	*
45	* @param offset is the value to insert.
46	*/
47	public void insertOffset(STextProcessorData procData, int offset) {
48	int index = count - 1; // index of greatest member <= offset
49	// look up after which member the new offset should be inserted
50	while (index >= 0) {
51	int wrkOffset = offsets[index];
52	if (offset > wrkOffset)
53	break;
54	if (offset == wrkOffset)
55	return; // avoid duplicates
56	index--;
57	}
58	index++; // index now points at where to insert
59	int length = count - index; // number of members to move up
60	if (length > 0) // shift right all members greater than offset
61	System.arraycopy(offsets, index, offsets, index + 1, length);
62	offsets[index] = offset;
63	count++; // number of used entries
64	// if the offset is 0, adding a mark does not change anything
65	if (offset < 1)
66	return;
67	STextCharTypes dirProps = procData.dirProps;
68	if (dirProps == null)
69	return;
70
71	byte dirProp = dirProps.getBidiTypeAt(offset);
72	// if the current char is a strong one or a digit, we change the
73	// dirProp of the previous char to account for the inserted mark.
74	if (dirProp == L \|\| dirProp == R \|\| dirProp == AL \|\| dirProp == EN \|\| dirProp == AN)
75	index = offset - 1;
76	else
77	// if the current char is a neutral, we change its own dirProp
78	index = offset;
79
80	dirProps.setBidiTypeAt(index, STRONGS[procData.direction]);
81	return;
82
83	}
84
85	/**
86	* Make sure that there is at least 3 free entries in the offsets array.
87	*/
88	public void ensureRoom() {
89	// make sure there are at least 3 empty slots in offsets
90	if ((offsets.length - count) < 3) {
91	int[] newOffsets = new int[offsets.length * 2];
92	System.arraycopy(offsets, 0, newOffsets, 0, count);
93	offsets = newOffsets;
94	}
95	}
96
97	/**
98	* Get all and only the used offset entries.
99	*
100	* @return the current used entries of the offsets array.
101	*/
102	public int[] getArray() {
103	if (count == offsets.length)
104	return offsets;
105	int[] array = new int[count];
106	System.arraycopy(offsets, 0, array, 0, count);
107	return array;
108	}
109	}

Added Link Here

(-)src/org/eclipse/equinox/bidi/custom/STextProcessorData.java (+52 lines)
1	package org.eclipse.equinox.bidi.custom;
2
3	import org.eclipse.equinox.bidi.STextEnvironment;
4	import org.eclipse.equinox.bidi.STextHandler;
5
6	/**
7	* This class regroups core information related to structured text operations.
8	* It is convenient to pass information between caller and called method.
9	*
10	* @author Matitiahu Allouche
11	*
12	*/
13	public class STextProcessorData {
14	/**
15	* The handler
16	*/
17	public STextHandler handler;
18	/**
19	* The processor
20	*/
21	public STextProcessor processor;
22	/**
23	* The environment
24	*/
25	public STextEnvironment environment;
26	/**
27	* The class instance to get and set the bidi type of characters
28	*/
29	public STextCharTypes dirProps;
30	/**
31	* The class instance to manipulate the array of offsets to
32	* directional control characters
33	*/
34	public STextOffsets offsets;
35	/**
36	* The source text
37	*/
38	public String text;
39	/**
40	* The base direction of the source text
41	*/
42	public int direction;
43	/**
44	* The orientation of the component which is to display the text
45	*/
46	public int orientation;
47	/**
48	* The length of the prefix (LRE, RLE, LRM or RLM) to be prepended
49	* to the text
50	*/
51	public int prefixLength;
52	}

Lines 1-25 Link Here

(-)src/org/eclipse/equinox/bidi/custom/package.html (-25 / +29 lines)
1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">	1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
2	<html>	2	<html>
3	<head>	3	<head>
4	<META name="Author" content="Matitiahu Allouche">	4	<META name="Author" content="Matitiahu Allouche">
5	</head>	5	</head>
6	<body bgcolor="white">	6	<body bgcolor="white">
7		7
8	This package provides classes for	8	This package provides classes for
9	developing structured text processors.	9	developing structured text processors.
10		10
11	<ul>	11	<ul>
12	<li>{@link <a href="STextProcessor.html">STextProcessor</a>}	12	<li>{@link <a href="STextProcessor.html">STextProcessor</a>}
13	is a generic processor to be used as superclass for specific	13	is a generic processor to be used as superclass for specific
14	processors.</li>	14	processors.</li>
15	<li>{@link <a href="STextStringProcessor.html">STextStringProcessor</a>}	15	<li>{@link <a href="STextProcessorData.html">STextProcessorData</a>}
16	is a class which allows retrieval of the defined processor types and of the	16	regroups data which can be used by processors.</li>
17	corresponding processors.</li>	17	<li>{@link <a href="STextCharTypes.html">STextStringCharTypes</a>}
18	</ul>	18	allows getting and setting the bidi class of characters.</li>
19		19	<li>{@link <a href="STextOffsets.html">STextStringOffsets</a>}
20	This package is to be used together with package	20	allows manipulation of offsets at which directional control
21	{@link <a href="..\package-summary.html">	21	characters are inserted.</li>
22	org.eclipse.equinox.bidi</a>}.	22	</ul>
23		23
24	</body>	24	This package is to be used together with package
25	</html>	25	{@link <a href="..\package-summary.html">
		26	org.eclipse.equinox.bidi</a>}.
		27
		28	</body>
		29	</html>

Lines 10-18 Link Here

(-)src/org/eclipse/equinox/bidi/internal/STextDelims.java (-8 / +7 lines)
10	******************************************************************************/	10	******************************************************************************/
11	package org.eclipse.equinox.bidi.internal;	11	package org.eclipse.equinox.bidi.internal;
12		12
13	import org.eclipse.equinox.bidi.STextEnvironment;
14	import org.eclipse.equinox.bidi.custom.STextCharTypes;
15	import org.eclipse.equinox.bidi.custom.STextProcessor;	13	import org.eclipse.equinox.bidi.custom.STextProcessor;
		14	import org.eclipse.equinox.bidi.custom.STextProcessorData;
16		15
17	/**	16	/**
18	* A base processor for structured text composed of text segments separated	17	* A base processor for structured text composed of text segments separated
Lines 44-52 Link Here
44	*	43	*
45	* @see #getDelimiters	44	* @see #getDelimiters
46	*/	45	*/
47	public int indexOfSpecial(STextEnvironment environment, String text, STextCharTypes dirProps, int[] offsets, int caseNumber, int fromIndex) {	46	public int indexOfSpecial(STextProcessorData procData, int caseNumber, int fromIndex) {
48	char delim = getDelimiters().charAt((caseNumber - 1) * 2);	47	char delim = getDelimiters().charAt((caseNumber - 1) * 2);
49	return text.indexOf(delim, fromIndex);	48	return procData.text.indexOf(delim, fromIndex);
50	}	49	}
51		50
52	/**	51	/**
Lines 59-71 Link Here
59	* @return the position after the matching end delimiter, or the length	58	* @return the position after the matching end delimiter, or the length
60	* of <code>text</code> if no end delimiter is found.	59	* of <code>text</code> if no end delimiter is found.
61	*/	60	*/
62	public int processSpecial(STextEnvironment environment, String text, STextCharTypes dirProps, int[] offsets, int[] state, int caseNumber, int separLocation) {	61	public int processSpecial(STextProcessorData procData, int caseNumber, int separLocation) {
63	STextProcessor.processSeparator(text, dirProps, offsets, separLocation);	62	STextProcessor.processSeparator(procData, separLocation);
64	int loc = separLocation + 1;	63	int loc = separLocation + 1;
65	char delim = getDelimiters().charAt((caseNumber * 2) - 1);	64	char delim = getDelimiters().charAt((caseNumber * 2) - 1);
66	loc = text.indexOf(delim, loc);	65	loc = procData.text.indexOf(delim, loc);
67	if (loc < 0)	66	if (loc < 0)
68	return text.length();	67	return procData.text.length();
69	return loc + 1;	68	return loc + 1;
70	}	69	}
71		70

Lines 1-183 Link Here

(-)src/org/eclipse/equinox/bidi/package.html (-183 / +191 lines)
1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">	1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
2	<html>	2	<html>
3	<head>	3	<head>
4	<META name="Author" content="Matitiahu Allouche">	4	<META name="Author" content="Matitiahu Allouche">
5	</head>	5	</head>
6	<body bgcolor="white">	6	<body bgcolor="white">
7		7
8	This package provides classes for	8	This package provides classes for
9	processing structured text.	9	processing structured text.
10	<p>	10	<p>
11	There are various types of structured text. Each type should	11	There are various types of structured text. Each type should
12	be handled by a specific <i>processor</i>. A number of standard	12	be handled by a specific <i>processor</i>. A number of standard
13	processors are supplied in the associated package	13	processors are supplied in the associated package
14	{@link <a href="internal\consumable\package-summary.html">	14	{@link <a href="internal\consumable\package-summary.html">
15	org.eclipse.equinox.bidi.internal.consumable</a>}.	15	org.eclipse.equinox.bidi.internal.consumable</a>}.
16		16
17	<h2>Introduction to Structured Text</h2>	17	<h2>Introduction to Structured Text</h2>
18	<p>	18	<p>
19	Bidirectional text offers interesting challenges to presentation systems.	19	Bidirectional text offers interesting challenges to presentation systems.
20	For plain text, the Unicode Bidirectional Algorithm	20	For plain text, the Unicode Bidirectional Algorithm
21	(<a href="http://www.unicode.org/reports/tr9/">UBA</a>)	21	(<a href="http://www.unicode.org/reports/tr9/">UBA</a>)
22	generally specifies satisfactorily how to reorder bidirectional text for	22	generally specifies satisfactorily how to reorder bidirectional text for
23	display. This algorithm is implemented in Java's presentation system.	23	display. This algorithm is implemented in Java's presentation system.
24	</p><p>	24	</p><p>
25	However, all bidirectional text is not necessarily plain text. There	25	However, all bidirectional text is not necessarily plain text. There
26	are also instances of text structured to follow a given syntax, which	26	are also instances of text structured to follow a given syntax, which
27	should be reflected in the display order. The general algorithm, which	27	should be reflected in the display order. The general algorithm, which
28	has no awareness of these special cases, often gives incorrect results	28	has no awareness of these special cases, often gives incorrect results
29	when displaying such structured text.	29	when displaying such structured text.
30	</p><p>	30	</p><p>
31	The general idea in handling structured text is to add directional	31	The general idea in handling structured text is to add directional
32	formatting characters at proper locations in the text to supplement the	32	formatting characters at proper locations in the text to supplement the
33	standard algorithm, so that the final result is correctly displayed	33	standard algorithm, so that the final result is correctly displayed
34	using the UBA.	34	using the UBA.
35	</p><p>	35	</p><p>
36	A class which handles structured text is thus essentially a	36	A class which handles structured text is thus essentially a
37	transformation engine which receives text without directional formatting	37	transformation engine which receives text without directional formatting
38	characters as input and produces as output the same text with added	38	characters as input and produces as output the same text with added
39	directional formatting characters, hopefully in the minimum quantity	39	directional formatting characters, hopefully in the minimum quantity
40	which is sufficient to ensure correct display, considering the type of	40	which is sufficient to ensure correct display, considering the type of
41	structured text involved.	41	structured text involved.
42	</p><p>	42	</p><p>
43	In this package, text without directional formatting characters is	43	In this package, text without directional formatting characters is
44	called <b><i>lean</i></b> text while the text with added directional	44	called <b><i>lean</i></b> text while the text with added directional
45	formatting characters is called <b><i>full</i></b> text.	45	formatting characters is called <b><i>full</i></b> text.
46	</p><p>	46	</p><p>
47	The class {@link	47	The class {@link
48	<a href="STextEngine.html"><b>STextEngine</b></a>}	48	<a href="STextHandler.html"><b>STextHandler</b></a>}
49	is the main tool for processing structured text. It facilitates	49	is the main tool for processing structured text. It facilitates
50	handling several types of structured text, each type being handled	50	handling several types of structured text, each type being handled
51	by a specific	51	by a specific
52	{@link <a href="custom\STextProcessor.html"><b><i>processor</i></b></a>} :</p>	52	{@link <a href="custom\STextProcessor.html"><b><i>processor</i></b></a>} :</p>
53	<ul>	53	<ul>
54	<li>property (name=value)</li>	54	<li>property (name=value)</li>
55	<li>compound name (xxx_yy_zzzz)</li>	55	<li>compound name (xxx_yy_zzzz)</li>
56	<li>comma delimited list</li>	56	<li>comma delimited list</li>
57	<li>system(user)</li>	57	<li>system(user)</li>
58	<li>directory and file path</li>	58	<li>directory and file path</li>
59	<li>e-mail address</li>	59	<li>e-mail address</li>
60	<li>URL</li>	60	<li>URL</li>
61	<li>regular expression</li>	61	<li>regular expression</li>
62	<li>Xpath</li>	62	<li>Xpath</li>
63	<li>Java code</li>	63	<li>Java code</li>
64	<li>SQL statements</li>	64	<li>SQL statements</li>
65	<li>RTL arithmetic expressions</li>	65	<li>RTL arithmetic expressions</li>
66	</ul>	66	</ul>
67	<p>	67	<p>
68	For each of these types, a static instance is defined in <b>STextEngine</b>.	68	For each of these types, a static instance is defined in
69	These pre-defined instances can be used as argument in the methods of	69	{@link <a href="STextProcessorFactory.html">STextProcessorFactory</a>}.
70	<b>STextEngine</b>.	70	These pre-defined instances can be used as argument when initiating a
71	</p><p>	71	<b>STextHandler</b>.
72	Other classes in this package may be used to	72	</p><p>
73	complement and facilitate the action of	73	Other classes in this package may be used to
74	{@link <a href="STextEngine.html">STextEngine</a>}:	74	complement and facilitate the action of
75	<ul>	75	{@link <a href="STextHandler.html">STextHandler</a>}:
76	<li>{@link <a href="STextEnvironment.html">STextEnvironment</a>}	76	<ul>
77	regroups details about the environment</li>.	77	<li>{@link <a href="STextEnvironment.html">STextEnvironment</a>}
78	<li>{@link <a href="STextUtil.html">STextUtil</a>}	78	regroups details about the environment.</li>
79	provides a number of convenience methods to process some common types of	79	<li>{@link <a href="STextProcessorFactory.html">STextProcessorFactory</a>}
80	structured text. When using methods in this class, there is no need	80	allows retrieval of the defined processor types and of the
81	to use other classes of this package. However, the other classes allow	81	corresponding processors.</li>
82	more precise control and possibly better performance.</li>	82	<li>{@link <a href="STextUtil.html">STextUtil</a>}
83	<li>{@link <a href="STextStringRecord.html">STextStringRecord</a>}	83	provides a number of convenience methods to process some common types of
84	allows to record strings which are structured text with the	84	structured text. When using methods in this class, there is no need
85	type of the appropriate processor,	85	to use other classes of this package. However, the other classes allow
86	so that another module can check if a given string has been recorded	86	more precise control and possibly better performance.</li>
87	as a structured text string and run its processor.</li>	87	<li>{@link <a href="STextStringRecord.html">STextStringRecord</a>}
88	</ul>	88	allows to record strings which are structured text with the
89	<p>	89	type of the appropriate processor,
90	{@link <a href="STextEngine.html">STextEngine</a>} and the	90	so that another module can check if a given string has been recorded
91	other classes mentioned above are intended for users who	91	as a structured text string and run its processor.</li>
92	need to process structured text for which there already exist	92	</ul>
93	processors.	93	<p>
94	<p>	94	{@link <a href="STextHandler.html">STextHandler</a>} and the
95	Developers who want to develop new processors to support types of structured text	95	other classes mentioned above are intended for users who
96	not currently supported can use the following components of the	96	need to process structured text for which there already exist
97	package {@link <a href="custom\package-summary.html">	97	processors.
98	org.eclipse.equinox.bidi.custom</a>}:	98	<p>
99	<ul>	99	Developers who want to develop new processors to support types of structured text
100	<li>{@link <a href="custom\STextProcessor.html">STextProcessor</a>}	100	not currently supported can use the following components of the
101	is a generic processor to be used as superclass for specific	101	package {@link <a href="custom\package-summary.html">
102	processors.</li>	102	org.eclipse.equinox.bidi.custom</a>}:
103	<li>{@link <a href="custom\STextStringProcessor.html">STextStringProcessor</a>}	103	<ul>
104	allows retrieval of the defined processor types and of the	104	<li>{@link <a href="custom\STextProcessor.html">STextProcessor</a>}
105	corresponding processors.</li>	105	is a generic processor to be used as superclass for specific
106	</ul>	106	processors.</li>
107	<p>	107	<li>{@link <a href="custom\STextProcessorData.html">STextProcessorData</a>}
108	There are two other packages associated with the current one:	108	regroups data which can be used by processors.</li>
109	<ul>	109	<li>{@link <a href="custom\STextCharTypes.html">STextStringCharTypes</a>}
110	<li>{@link <a href="internal\package-summary.html">	110	allows getting and setting the bidi class of characters.</li>
111	org.eclipse.equinox.bidi.internal</a>}</li>	111	<li>{@link <a href="custom\STextOffsets.html">STextStringOffsets</a>}
112	<li>{@link <a href="internal\consumable\package-summary.html">	112	allows manipulation of offsets at which directional control
113	org.eclipse.equinox.bidi.internal.consumable</a>}</li>	113	characters are inserted.</li>
114	</ul>	114	</ul>
115	The tools in the first package can serve as example of how to develop	115	<p>
116	processors for currently unsupported types of structured text.<br>	116	There are two other packages associated with the current one:
117	The latter package contains classes for the processors implementing	117	<ul>
118	the currently supported types of structured text.	118	<li>{@link <a href="internal\package-summary.html">
119	<p>	119	org.eclipse.equinox.bidi.internal</a>}</li>
120	However, users wishing to process the currently supported types of	120	<li>{@link <a href="internal\consumable\package-summary.html">
121	structured text typically don't need to interact with these	121	org.eclipse.equinox.bidi.internal.consumable</a>}</li>
122	two packages.	122	</ul>
123		123	The tools in the first package can serve as example of how to develop
124	<p> </p>	124	processors for currently unsupported types of structured text.<br>
125		125	The latter package contains classes for the processors implementing
126	<h2>Abbreviations used in the documentation of this package</h2>	126	the currently supported types of structured text.
127		127	<p>
128	<dl>	128	However, users wishing to process the currently supported types of
129	<dt><b>UBA</b><dd>Unicode Bidirectional Algorithm	129	structured text typically don't need to interact with these
130		130	two packages.
131	<dt><b>Bidi</b><dd>Bidirectional	131
132		132	<p> </p>
133	<dt><b>GUI</b><dd>Graphical User Interface	133
134		134	<h2>Abbreviations used in the documentation of this package</h2>
135	<dt><b>UI</b><dd>User Interface	135
136		136	<dl>
137	<dt><b>LTR</b><dd>Left to Right	137	<dt><b>UBA</b><dd>Unicode Bidirectional Algorithm
138		138
139	<dt><b>RTL</b><dd>Right to Left	139	<dt><b>Bidi</b><dd>Bidirectional
140		140
141	<dt><b>LRM</b><dd>Left-to-Right Mark	141	<dt><b>GUI</b><dd>Graphical User Interface
142		142
143	<dt><b>RLM</b><dd>Right-to-Left Mark	143	<dt><b>UI</b><dd>User Interface
144		144
145	<dt><b>LRE</b><dd>Left-to-Right Embedding	145	<dt><b>LTR</b><dd>Left to Right
146		146
147	<dt><b>RLE</b><dd>Right-to-Left Embedding	147	<dt><b>RTL</b><dd>Right to Left
148		148
149	<dt><b>PDF</b><dd>Pop Directional Formatting	149	<dt><b>LRM</b><dd>Left-to-Right Mark
150	</dl>	150
151		151	<dt><b>RLM</b><dd>Right-to-Left Mark
152	<p> </p>	152
153		153	<dt><b>LRE</b><dd>Left-to-Right Embedding
154	<h2>Known Limitations</h2>	154
155		155	<dt><b>RLE</b><dd>Right-to-Left Embedding
156	<p>The proposed solution is making extensive usage of LRM, RLM, LRE, RLE	156
157	and PDF directional controls which are invisible but affect the way bidi	157	<dt><b>PDF</b><dd>Pop Directional Formatting
158	text is displayed. The following related key points merit special	158	</dl>
159	attention:</p>	159
160		160	<p> </p>
161	<ul>	161
162	<li>Implementations of the UBA on various platforms (e.g., Windows and	162	<h2>Known Limitations</h2>
163	Linux) are very similar but nevertheless have known differences. Those	163
164	differences are minor and will not have a visible effect in most cases.	164	<p>The proposed solution is making extensive usage of LRM, RLM, LRE, RLE
165	However there might be cases in which the same bidi text on two	165	and PDF directional controls which are invisible but affect the way bidi
166	platforms might look different. These differences will surface in Java	166	text is displayed. The following related key points merit special
167	applications when they use the platform visual components for their UI	167	attention:</p>
168	(e.g., AWT, SWT).</li>	168
169		169	<ul>
170	<li>It is assumed that the presentation engine supports LRE, RLE and	170	<li>Implementations of the UBA on various platforms (e.g., Windows and
171	PDF directional formatting characters.</li>	171	Linux) are very similar but nevertheless have known differences. Those
172		172	differences are minor and will not have a visible effect in most cases.
173	<li>Because some presentation engines are not strictly conformant to the	173	However there might be cases in which the same bidi text on two
174	UBA, the implementation of structured text in this package adds LRM	174	platforms might look different. These differences will surface in Java
175	or RLM characters in association with LRE, RLE or PDF in cases where	175	applications when they use the platform visual components for their UI
176	this would not be needed if the presentation engine was fully conformant	176	(e.g., AWT, SWT).</li>
177	to the UBA. Such added marks will not have harmful effects on	177
178	conformant presentation engines and will help less conformant engines to	178	<li>It is assumed that the presentation engine supports LRE, RLE and
179	achieve the desired presentation.</li>	179	PDF directional formatting characters.</li>
180	</ul>	180
181		181	<li>Because some presentation engines are not strictly conformant to the
182	</body>	182	UBA, the implementation of structured text in this package adds LRM
183	</html>	183	or RLM characters in association with LRE, RLE or PDF in cases where
		184	this would not be needed if the presentation engine was fully conformant
		185	to the UBA. Such added marks will not have harmful effects on
		186	conformant presentation engines and will help less conformant engines to
		187	achieve the desired presentation.</li>
		188	</ul>
		189
		190	</body>
		191	</html>