Examples of com.ibm.icu.text.Normalizer2

com.ibm.icu.text.Normalizer2
nicode.org/unicode/reports/tr15/ However, this API supports additional normalization forms for specialized purposes. For example, NFKC_Casefold is provided via getInstance("nfkc_cf", COMPOSE) and can be used in implementations of UTS #46.
Not only are the standard compose and decompose modes supplied, but additional modes are provided as documented in the Mode enum.
Some of the functions in this class identify normalization boundaries. At a normalization boundary, the portions of the string before it and starting from it do not interact and can be handled independently.
The spanQuickCheckYes() stops at a normalization boundary. When the goal is a normalized string, then the text before the boundary can be copied, and the remainder can be processed with normalizeSecondAndAppend().
The hasBoundaryBefore(), hasBoundaryAfter() and isInert() functions test whether a character is guaranteed to be at a normalization boundary, regardless of context. This is used for moving from one normalization boundary to the next or preceding boundary, and for performing iterative normalization.
Iterative normalization is useful when only a small portion of a longer string needs to be processed. For example, in ICU, iterative normalization is used by the NormalizationTransliterator (to avoid replacing already-normalized text) and ucol_nextSortKeyPart() (to process only the substring for which sort key bytes are computed).
The set of normalization boundaries returned by these functions may not be complete: There may be more boundaries that could be returned. Different functions may return different boundaries. @stable ICU 4.4 @author Markus W. Scherer


public class TestICUNormalizer2CharFilter extends BaseTokenStreamTestCase {


  public void testNormalization() throws IOException {
    String input = "ʰ㌰゙5℃№㈱㌘，バッファーの正規化のテスト．㋐㋑㋒㋓㋔ｶｷｸｹｺｻﾞｼﾞｽﾞｾﾞｿﾞg̈각/각நிเกषिchkʷक्षि";
    Normalizer2 normalizer = Normalizer2.getInstance(null, "nfkc_cf", Normalizer2.Mode.COMPOSE);
    String expectedOutput = normalizer.normalize(input);


    CharFilter reader = new ICUNormalizer2CharFilter(new StringReader(input), normalizer);
    char[] tempBuff = new char[10];
    StringBuilder output = new StringBuilder();
    while (true) {
      int length = reader.read(tempBuff);
      if (length == -1) {
        break;
      }
      output.append(tempBuff, 0, length);
      assertEquals(output.toString(), normalizer.normalize(input.substring(0, reader.correctOffset(output.length()))));
    }


    assertEquals(expectedOutput, output.toString());
  }

View Full Code Here

  /** Creates a new ICUNormalizer2CharFilterFactory */
  public ICUNormalizer2CharFilterFactory(Map<String,String> args) {
    super(args);
    String name = get(args, "name", "nfkc_cf");
    String mode = get(args, "mode", Arrays.asList("compose", "decompose"), "compose");
    Normalizer2 normalizer = Normalizer2.getInstance
        (null, name, "compose".equals(mode) ? Normalizer2.Mode.COMPOSE : Normalizer2.Mode.DECOMPOSE);
    
    String filter = get(args, "filter");
    if (filter != null) {
      UnicodeSet set = new UnicodeSet(filter);

View Full Code Here

  /** Creates a new ICUNormalizer2FilterFactory */
  public ICUNormalizer2FilterFactory(Map<String,String> args) {
    super(args);
    String name = get(args, "name", "nfkc_cf");
    String mode = get(args, "mode", Arrays.asList("compose", "decompose"), "compose");
    Normalizer2 normalizer = Normalizer2.getInstance
        (null, name, "compose".equals(mode) ? Normalizer2.Mode.COMPOSE : Normalizer2.Mode.DECOMPOSE);
    
    String filter = get(args, "filter");
    if (filter != null) {
      UnicodeSet set = new UnicodeSet(filter);

View Full Code Here


public class TestICUNormalizer2CharFilter extends BaseTokenStreamTestCase {


  public void testNormalization() throws IOException {
    String input = "ʰ㌰゙5℃№㈱㌘，バッファーの正規化のテスト．㋐㋑㋒㋓㋔ｶｷｸｹｺｻﾞｼﾞｽﾞｾﾞｿﾞg̈각/각நிเกषिchkʷक्षि";
    Normalizer2 normalizer = Normalizer2.getInstance(null, "nfkc_cf", Normalizer2.Mode.COMPOSE);
    String expectedOutput = normalizer.normalize(input);


    CharFilter reader = new ICUNormalizer2CharFilter(new StringReader(input), normalizer);
    char[] tempBuff = new char[10];
    StringBuilder output = new StringBuilder();
    while (true) {
      int length = reader.read(tempBuff);
      if (length == -1) {
        break;
      }
      output.append(tempBuff, 0, length);
      assertEquals(output.toString(), normalizer.normalize(input.substring(0, reader.correctOffset(output.length()))));
    }


    assertEquals(expectedOutput, output.toString());
  }

View Full Code Here

TOP

Related Classes of com.ibm.icu.text.Normalizer2

com.ibm.icu.impl.Norm2AllModes

org.apache.lucene.analysis.icu.ICUNormalizer2CharFilterFactory

org.apache.lucene.analysis.icu.ICUNormalizer2FilterFactory

org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.