Examples of com.ibm.icu.text.UnicodeSet$ScriptExtensionsFilter

com.ibm.icu.text.UnicodeSet

cu-project.org/userguide/unicodeSet.html"> http://www.icu-project.org/userguide/unicodeSet.html. Actual determination of property data is defined by the underlying Unicode database as implemented by UCharacter.

Patterns specify individual characters, ranges of characters, and Unicode property sets. When elements are concatenated, they specify their union. To complement a set, place a '^' immediately after the opening '['. Property patterns are inverted by modifying their delimiters; "[:^foo]" and "\P{foo}". In any other location, '^' has no special meaning.

Ranges are indicated by placing two a '-' between two characters, as in "a-z". This specifies the range of all characters from the left to the right, in Unicode order. If the left character is greater than or equal to the right character it is a syntax error. If a '-' occurs as the first character after the opening '[' or '[^', or if it occurs as the last character before the closing ']', then it is taken as a literal. Thus "[a\\-b]", "[-ab]", and "[ab-]" all indicate the same set of three characters, 'a', 'b', and '-'.

Sets may be intersected using the '&' operator or the asymmetric set difference may be taken using the '-' operator, for example, "[[:L:]&[\\u0000-\\u0FFF]]" indicates the set of all Unicode letters with values less than 4096. Operators ('&' and '|') have equal precedence and bind left-to-right. Thus "[[:L:]-[a-z]-[\\u0100-\\u01FF]]" is equivalent to "[[[:L:]-[a-z]]-[\\u0100-\\u01FF]]". This only really matters for difference; intersection is commutative.

`[a]`	The set containing 'a'
`[a-z]`	The set containing 'a' through 'z' and all letters in between, in Unicode order
`[^a-z]`	The set containing all characters but 'a' through 'z', that is, U+0000 through 'a'-1 and 'z'+1 through U+10FFFF
`[[pat1][pat2]]`	The union of sets specified by pat1 and pat2
`[[pat1]&[pat2]]`	The intersection of sets specified by pat1 and pat2
`[[pat1]-[pat2]]`	The asymmetric difference of sets specified by pat1 and pat2
`[:Lu:] or \p{Lu}`	The set of characters having the specified Unicode property; in this case, Unicode uppercase letters
`[:^Lu:] or \P{Lu}`	The set of characters not having the given Unicode property

Warning: you cannot add an empty string ("") to a UnicodeSet.

Formal syntax

pattern := ('[' '^'? item* ']') | property

item := char | (char '-' char) | pattern-expr

pattern-expr := pattern | pattern-expr pattern | pattern-expr op pattern

op := '&' | '-'

special := '[' | ']' | '-'

char := any character that is notspecial | ('\\'any character) | ('\u' hex hex hex hex)

hex := any character for which Character.digit(c, 16) returns a non-negative result

property := a Unicode property set pattern

Legend:

a := b a may be replaced by b

a? zero or one instance of a

a* one or more instances of a

a | b either a or b

'a' the literal string between the quotes

To iterate over contents of UnicodeSet, use UnicodeSetIterator class. @author Alan Liu @stable ICU 2.0 @see UnicodeSetIterator

        "[abc]","[A-Ca-c]",
        CASE,
        "[ABC]","[A-Ca-c]",
    };
    
    UnicodeSet s = new UnicodeSet();
    UnicodeSet t = new UnicodeSet();
    for (int i=0; i<DATA.length; i+=3) {
      int selector = Integer.parseInt(DATA[i]);
      String pat = DATA[i+1];
      String exp = DATA[i+2];
      s.applyPattern(pat);
      s.closeOver(selector);
      t.applyPattern(exp);
      if (s.equals(t)) {
        logln("Ok: " + pat + ".closeOver(" + selector + ") => " + exp);
      } else {
        errln("FAIL: " + pat + ".closeOver(" + selector + ") => " +
            s.toPattern(true) + ", expected " + exp);
      }
    }
    
    // Test the pattern API
    s.applyPattern("[abc]", UnicodeSet.CASE);
    expectContainment(s, "abcABC", "defDEF");
    s = new UnicodeSet("[^abc]", UnicodeSet.CASE);
    expectContainment(s, "defDEF", "abcABC");
  }

View Full Code Here

        pat = Utility.unescape(pat);
      }
      // Pattern is only good for pass 1
      boolean isPatternValid = (pass==1);
      
      UnicodeSet set = null;
      try {
        set = new UnicodeSet(pat);
      } catch (IllegalArgumentException e) {
        set = null;
      }
      if ((set != null) != isPatternValid){
        errln("FAIL: applyPattern(" +
            Utility.escape(pat) + ") => " + set);
        continue;
      }
      if (set == null) {
        continue;
      }
      if (set.contains((char)0x0644)){
        errln("FAIL: " + Utility.escape(pat) + " contains(U+0664)");
      }
      
      String newpat = set.toPattern(true);
      if (newpat.equals(exp)) {
        logln(Utility.escape(pat) + " => " + newpat);
      } else {
        errln("FAIL: " + Utility.escape(pat) + " => " + newpat);
      }
      
      for (int i=0; i<set.getRangeCount(); ++i) {
        StringBuffer str = new StringBuffer("Range ");
        str.append((char)(0x30 + i))
        .append(": ");
        UTF16.append(str, set.getRangeStart(i));
        str.append(" - ");
        UTF16.append(str, set.getRangeEnd(i));
        String s = Utility.escape(str.toString() + " (" + set.getRangeStart(i) + " - " +
            set.getRangeEnd(i) + ")");
        if (set.getRangeStart(i) < 0) {
          errln("FAIL: " + s);
        } else {
          logln(s);
        }
      }

View Full Code Here

      // Input pattern and expected output pattern
      String inpat = DATA[i], exppat = DATA[i+1];
      i += 2;
      
      ParsePosition pos = new ParsePosition(0);
      UnicodeSet us = new UnicodeSet(inpat, pos, sym);
      
      // results
      if (pos.getIndex() != inpat.length()) {
        errln("Failed to read to end of string \""
            + inpat + "\": read to "
            + pos.getIndex() + ", length is "
            + inpat.length());
      }
      
      UnicodeSet us2 = new UnicodeSet(exppat);
      if (!us.equals(us2)) {
        errln("Failed, got " + us + ", expected " + us2);
      } else {
        logln("Ok, got " + us);
      }
      
      //cover Unicode(String,ParsePosition,SymbolTable,int)
      ParsePosition inpos = new ParsePosition(0);
      UnicodeSet inSet = new UnicodeSet(inpat, inpos, sym, UnicodeSet.IGNORE_SPACE);
      UnicodeSet expSet = new UnicodeSet(exppat);
      if (!inSet.equals(expSet)) {
        errln("FAIL: Failed, got " + inSet + ", expected " + expSet);
      } else {
        logln("OK: got " + inSet);
      }

View Full Code Here

    expectEqual("POSIX graph", "[:graph:]", "[^\\p{Whitespace}\\p{Control}\\p{Surrogate}\\p{Unassigned}]");
    expectEqual("POSIX print", "[:print:]", "[[:graph:][:blank:]-[\\p{Control}]]");
  }
  
  public void TestHangulSyllable() {
      final UnicodeSet lvt = new UnicodeSet("[:Hangul_Syllable_Type=LVT_Syllable:]");
      assertNotEquals("LVT count", new UnicodeSet(), lvt);
      logln(lvt + ": " + lvt.size());
      final UnicodeSet lv = new UnicodeSet("[:Hangul_Syllable_Type=LV_Syllable:]");
      assertNotEquals("LV count", new UnicodeSet(), lv);
      logln(lv + ": " + lv.size());
  }

View Full Code Here

  
  /**
   * Test that frozen classes disallow changes. For 4217
   */
  public void TestFrozen() {
    UnicodeSet test = new UnicodeSet("[[:whitespace:]A]");
    test.freeze();
    checkModification(test, true);
    checkModification(test, false);
  }

View Full Code Here

  }
  
  public void checkModification(UnicodeSet original, boolean isFrozen) {
    main:
      for (int i = 0; ;++i) {
        UnicodeSet test = (UnicodeSet) (isFrozen ? original.clone() : original.cloneAsThawed());
        boolean gotException = true;
        boolean checkEquals = true;
        try {
          switch(i) {
            case 0: test.add(0); break;
            case 1: test.add(0,1); break;
            case 2: test.add("a"); break;
            case 3: List a = new ArrayList(); a.add("a"); test.addAll(a); break;
            case 4: test.addAll("ab"); break;
            case 5: test.addAll(new UnicodeSet("[ab]")); break;
            case 6: test.applyIntPropertyValue(0,0); break;
            case 7: test.applyPattern("[ab]"); break;
            case 8: test.applyPattern("[ab]", true); break;
            case 9: test.applyPattern("[ab]", 0); break;
            case 10: test.applyPropertyAlias("hex","true"); break;
            case 11: test.applyPropertyAlias("hex", "true", null); break;
            case 12: test.closeOver(UnicodeSet.CASE); break;
            case 13: test.compact(); checkEquals = false; break;
            case 14: test.complement(0); break;
            case 15: test.complement(0,0); break;
            case 16: test.complement("ab"); break;
            case 17: test.complementAll("ab"); break;
            case 18: test.complementAll(new UnicodeSet("[ab]")); break;
            case 19: test.remove(' '); break;
            case 20: test.remove(' ','a'); break;
            case 21: test.remove(" "); break;
            case 22: test.removeAll(" a"); break;
            case 23: test.removeAll(new UnicodeSet("[\\ a]")); break;
            case 24: test.retain(' '); break;
            case 25: test.retain(' ','a'); break;
            case 26: test.retain(" "); break;
            case 27: test.retainAll(" a"); break;
            case 28: test.retainAll(new UnicodeSet("[\\ a]")); break;
            case 29: test.set(0,1); break;
            case 30: test.set(new UnicodeSet("[ab]")); break;
            
            default: continue main; // so we don't keep having to change the endpoint, and gaps are not skipped.
            case 35: return;
          }
          gotException = false;
        } catch (UnsupportedOperationException e) {
          // do nothing
        }
        if (isFrozen && !gotException) errln(i + ") attempt to modify frozen object didn't result in an exception");
        if (!isFrozen && gotException) errln(i + ") attempt to modify thawed object did result in an exception");
        if (checkEquals) {
          if (test.equals(original)) {
            if (!isFrozen) errln(i + ") attempt to modify thawed object didn't change the object");
          } else { // unequal
            if (isFrozen) errln(i + ") attempt to modify frozen object changed the object");
          }
        }

View Full Code Here

    try{
      PrettyPrinter pp = new PrettyPrinter();
      
      int i = 0;
      for (; i < prettyData.length; ++i) {
        UnicodeSet test = new UnicodeSet(prettyData[i]);
        checkPrettySet(pp, i, test);
      }
      Random random = new Random(0);
      UnicodeSet test = new UnicodeSet();
      
      // To keep runtimes under control, make the number of random test cases
      //   to try depends on the test framework exhaustive setting.
      //  params.inclusions = 5:   default exhaustive value
      //  params.inclusions = 10:  max exhaustive value.
      int iterations = 50;
      if (params.inclusion > 5) {
        iterations = (params.inclusion-5) * 200;
      }
      for (; i < iterations; ++i) {
        double start = random.nextGaussian() * 0x10000;
        if (start < 0) start = - start;
        if (start > 0x10FFFF) {
          start = 0x10FFFF;
        }
        double end = random.nextGaussian() * 0x100;
        if (end < 0) end = -end;
        end = start + end;
        if (end > 0x10FFFF) {
          end = 0x10FFFF;
        }
        test.complement((int)start, (int)end);
        checkPrettySet(pp, i, test);
      }
    }catch(RuntimeException ex){
      warnln("Could not load Collator");
    }

View Full Code Here

    }
  }
  
  private void checkPrettySet(PrettyPrinter pp, int i, UnicodeSet test) {
    String pretty = pp.toPattern(test);
    UnicodeSet retry = new UnicodeSet(pretty);
    if (!test.equals(retry)) {
      errln(i + ". Failed test: " + test + " != " + pretty);
    } else {
      logln(i + ". Worked for " + truncate(test.toString()) + " => " + truncate(pretty));
    }

View Full Code Here

        "[abc\uD800\uDC00]",
        "[abc\\U00010000]",
    };
    for (int i=0; i<DATA.length; ++i) {
      logln("Test pattern " + i + " :" + Utility.escape(DATA[i]));
      UnicodeSet set = new UnicodeSet(DATA[i]);
      expectContainment(set,
          CharsToUnicodeString("abc\\U00010000"),
      "\uD800;\uDC00"); // split apart surrogate-pair
      if (set.size() != 4) {
        errln(Utility.escape("FAIL: " + DATA[i] + ".size() == " + 
            set.size() + ", expected 4"));
      }
    }
  }

View Full Code Here

  
  public void TestContains() {
    int limit = 256; // combinations to test
    for (int i = 0; i < limit; ++i) {
      logln("Trying: " + i);
      UnicodeSet x = bitsToSet(i);
      for (int j = 0; j < limit; ++j) {
        UnicodeSet y = bitsToSet(j);
        boolean containsNone = (i & j) == 0;
        boolean containsAll = (i & j) == j;
        boolean equals = i == j;
        if (containsNone != x.containsNone(y)) {
          x.containsNone(y); // repeat for debugging

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of com.ibm.icu.text.UnicodeSet$ScriptExtensionsFilter

com.ibm.icu.charset.CharsetISCII

com.ibm.icu.charset.CharsetSelector

com.ibm.icu.dev.demo.translit.CaseIterator

com.ibm.icu.dev.demo.translit.TransliterationChart

com.ibm.icu.dev.test.charset.TestConversion

com.ibm.icu.dev.test.charset.TestSelection

com.ibm.icu.dev.test.collator.RandomCollator

com.ibm.icu.dev.test.format.DateTimeGeneratorTest

com.ibm.icu.dev.test.lang.UCharacterCaseTest

com.ibm.icu.dev.test.lang.UCharacterTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

`pattern :=`	`('[' '^'? item* ']') \| property`
`item :=`	`char \| (char '-' char) \| pattern-expr`
`pattern-expr :=`	`pattern \| pattern-expr pattern \| pattern-expr op pattern`
`op :=`	`'&' \| '-'`
`special :=`	`'[' \| ']' \| '-'`
`char :=`	any character that is not`special \| ('\\'`any character`) \| ('\u' hex hex hex hex)`
`hex :=`	any character for which `Character.digit(c, 16)` returns a non-negative result
`property :=`	a Unicode property set pattern