Examples of PatternTokenizer

com.ibm.icu.impl.PatternTokenizer
A simple parsing class for patterns and rules. Handles '...' quotations, \\uxxxx and \\Uxxxxxxxx, and symple syntax. The '' (two quotes) is treated as a single quote, inside or outside a quote
- Any ignorable characters are ignored in parsing.
- Any syntax characters are broken into separate tokens
- Quote characters can be specified: '...', "...", and \x
- Other characters are treated as literals
org.apache.lucene.analysis.pattern.PatternTokenizer
This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".
- "pattern" is the regular expression.
- "group" says which group to extract into tokens.
group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): {@link String#split(java.lang.String)}

Using group >= 0 selects the matching group as the token. For example, if you have:
```
 pattern = \'([^\']+)\' group = 0 input = aaa 'bbb' 'ccc' 
```
the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.
@see Pattern
org.apache.solr.analysis.PatternTokenizer
This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".
- "pattern" is the regular expression.
- "group" says which group to extract into tokens.
group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): {@link String#split(java.lang.String)}

Using group >= 0 selects the matching group as the token. For example, if you have:
```
 pattern = \'([^\']+)\' group = 0 input = aaa 'bbb' 'ccc' 
```
the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.
@version $Id: PatternTokenizer.java 940806 2010-05-04 11:18:46Z uschindler $ @see Pattern

Examples of com.ibm.icu.impl.PatternTokenizer

        format.setTimeZone(rootZone);
    }


    public void TestPatternParser() {
        StringBuffer buffer = new StringBuffer();
        PatternTokenizer pp = new PatternTokenizer()
        .setIgnorableCharacters(new UnicodeSet("[-]"))
        .setSyntaxCharacters(new UnicodeSet("[a-zA-Z]"))
        .setEscapeCharacters(new UnicodeSet("[b#]"))
        .setUsingQuote(true);
        logln("Using Quote");
        for (int i = 0; i < patternTestData.length; ++i) {
            String patternTest = (String) patternTestData[i];
            CheckPattern(buffer, pp, patternTest);
        }
        String[] randomSet = {"abcdef", "$12!@#-", "'\\"};
        for (int i = 0; i < RANDOM_COUNT; ++i) {
            String patternTest = getRandomString(randomSet, 0, 10);
            CheckPattern(buffer, pp, patternTest);
        }
        logln("Using Backslash");
        pp.setUsingQuote(false).setUsingSlash(true);
        for (int i = 0; i < patternTestData.length; ++i) {
            String patternTest = (String) patternTestData[i];
            CheckPattern(buffer, pp, patternTest);
        }
        for (int i = 0; i < RANDOM_COUNT; ++i) {

View Full Code Here

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

                return "pattern";
            }


            @Override public Tokenizer create(Reader reader) {
                try {
                    return new PatternTokenizer(reader, Regex.compile("\\W+", null), -1);
                } catch (IOException e) {
                    throw new ElasticSearchIllegalStateException("failed to parse default pattern");
                }
            }
        }));

View Full Code Here

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

        this.group = settings.getAsInt("group", -1);
    }


    @Override public Tokenizer create(Reader reader) {
        try {
            return new PatternTokenizer(reader, pattern, group);
        } catch (IOException e) {
            throw new ElasticSearchIllegalStateException("failed to create pattern tokenizer", e);
        }
    }

View Full Code Here

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

   * Split the input using configured pattern
   */
  @Override
  public Tokenizer create(final Reader in) {
    try {
      return new PatternTokenizer(in, pattern, group);
    } catch( IOException ex ) {
      throw new RuntimeException("IOException thrown creating PatternTokenizer instance", ex);
    }
  }

View Full Code Here

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

        this.stopWords = stopWords;
    }


    @Override
    protected TokenStreamComponents createComponents(String s) {
        final Tokenizer tokenizer = new PatternTokenizer(pattern, -1);
        TokenStream stream = tokenizer;
        if (lowercase) {
            stream = new LowerCaseFilter(stream);
        }
        if (stopWords != null) {

View Full Code Here

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

        this.group = settings.getAsInt("group", -1);
    }


    @Override
    public Tokenizer create() {
        return new PatternTokenizer(pattern, group);
    }

View Full Code Here

Examples of org.apache.solr.analysis.PatternTokenizer

    Analyzer analyzer = new Analyzer() {
      @Override
      public TokenStream tokenStream(String fieldName, Reader reader) {
        TokenStream result = null;
        try {
          result = new PatternTokenizer(reader, pattern, 0);
        } catch (IOException e) {
        }
        return result;
      }
    };

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.