Examples of LowerCaseTokenizer

• org.apache.lucene.analysis.LowerCaseTokenizer
  LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

  Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

• org.apache.lucene.analysis.core.LowerCaseTokenizer
  LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

  Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

  You must specify the required {@link Version} compatibility when creating{@link LowerCaseTokenizer}:

  • As of 3.1, {@link CharTokenizer} uses an int based API to normalize anddetect token characters. See {@link CharTokenizer#isTokenChar(int)} and{@link CharTokenizer#normalize(int)} for details.

• uk.ac.ucl.panda.utility.parser.LowerCaseTokenizer
  LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

  Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Examples of uk.ac.ucl.panda.utility.parser.LowerCaseTokenizer