Examples of it.unibz.instasearch.indexing.tokenizers.standard.StandardTokenizer

it.unibz.instasearch.indexing.tokenizers.standard.StandardTokenizer

A grammar-based tokenizer constructed with JavaCC.

This should be a good tokenizer for most European-language documents:

Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

  
  @Override
  public TokenStream tokenStream(String fieldName, Reader reader)
  {
    if( Field.CONTENTS.toString().equals(fieldName) ) {
      TokenStream result = new StandardTokenizer(reader); // splits at ". ", "-"
      
      result = new WordSplitTokenizer(result);   // non-alphanumerics
      result = new DotSplitTokenizer(result);   // com.package.names
      result = new CamelCaseTokenizer(result);   // CamelCaseIdentifiers

    this.minWordLength = minWordLength;
  }
  
  public TokenStream tokenStream(Reader reader) {
    
    TokenStream result = new StandardTokenizer(reader); // splits at ". ", etc.
    
    // result = new SysoFilter(result);


    result = new WordSplitTokenizer(result);   // non-alphanumerics
    result = new DotSplitTokenizer(result);   // all.package.names, hyphen-separated-words

Examples of it.unibz.instasearch.indexing.tokenizers.standard.StandardTokenizer

Related Classes of it.unibz.instasearch.indexing.tokenizers.standard.StandardTokenizer