Examples of ArabicLetterTokenizer

org.apache.lucene.analysis.ar.ArabicLetterTokenizer
Tokenizer that breaks text into runs of letters and diacritics.
The problem with the standard Letter tokenizer is that it fails on diacritics. Handling similar to this is necessary for Indic Scripts, Hebrew, Thaana, etc.

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

   *         {@link ArabicNormalizationFilter},
   *         {@link PersianNormalizationFilter} and Persian Stop words
   */
  @Override
  public TokenStream tokenStream(String fieldName, Reader reader) {
    TokenStream result = new ArabicLetterTokenizer(reader);
    result = new LowerCaseFilter(result);
    result = new ArabicNormalizationFilter(result);
    /* additional persian-specific normalization */
    result = new PersianNormalizationFilter(result);
    /*

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

  public TokenStream reusableTokenStream(String fieldName, Reader reader)
      throws IOException {
    SavedStreams streams = (SavedStreams) getPreviousTokenStream();
    if (streams == null) {
      streams = new SavedStreams();
      streams.source = new ArabicLetterTokenizer(reader);
      streams.result = new LowerCaseFilter(streams.source);
      streams.result = new ArabicNormalizationFilter(streams.result);
      /* additional persian-specific normalization */
      streams.result = new PersianNormalizationFilter(streams.result);
      /*

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

    assureMatchVersion();
    warnDeprecated("Use StandardTokenizerFactory instead.");
  }


  public ArabicLetterTokenizer create(Reader input) {
    return new ArabicLetterTokenizer(luceneMatchVersion, input);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

   *         filtered with {@link LowerCaseFilter}, 
   *         {@link ArabicNormalizationFilter},
   *         {@link PersianNormalizationFilter} and Persian Stop words
   */
  public TokenStream tokenStream(String fieldName, Reader reader) {
    TokenStream result = new ArabicLetterTokenizer(reader);
    result = new LowerCaseFilter(result);
    result = new ArabicNormalizationFilter(result);
    /* additional persian-specific normalization */
    result = new PersianNormalizationFilter(result);
    /*

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

  public TokenStream reusableTokenStream(String fieldName, Reader reader)
      throws IOException {
    SavedStreams streams = (SavedStreams) getPreviousTokenStream();
    if (streams == null) {
      streams = new SavedStreams();
      streams.source = new ArabicLetterTokenizer(reader);
      streams.result = new LowerCaseFilter(streams.source);
      streams.result = new ArabicNormalizationFilter(streams.result);
      /* additional persian-specific normalization */
      streams.result = new PersianNormalizationFilter(streams.result);
      /*

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

 *
 **/
public class ArabicLetterTokenizerFactory extends BaseTokenizerFactory{


  public ArabicLetterTokenizer create(Reader input) {
    return new ArabicLetterTokenizer(input);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

    super.init(args);
    assureMatchVersion();
  }


  public ArabicLetterTokenizer create(Reader input) {
    return new ArabicLetterTokenizer(luceneMatchVersion, input);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

      Reader reader) {
    final Tokenizer source;
    if (matchVersion.onOrAfter(Version.LUCENE_3_1)) {
      source = new StandardTokenizer(matchVersion, reader);
    } else {
      source = new ArabicLetterTokenizer(matchVersion, reader);
    }
    TokenStream result = new LowerCaseFilter(matchVersion, source);
    result = new ArabicNormalizationFilter(result);
    /* additional persian-specific normalization */
    result = new PersianNormalizationFilter(result);

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

  public void testHehGoal() throws IOException {
    check("زادہ", "زاده");
  }


  private void check(final String input, final String expected) throws IOException {
    ArabicLetterTokenizer tokenStream = new ArabicLetterTokenizer(TEST_VERSION_CURRENT, 
        new StringReader(input));
    PersianNormalizationFilter filter = new PersianNormalizationFilter(
        tokenStream);
    assertTokenStreamContents(filter, new String[]{expected});
  }

View Full Code Here

Examples of org.apache.lucene.analysis.ar.ArabicLetterTokenizer

import org.apache.lucene.analysis.ar.ArabicLetterTokenizer;


public class ArabicLetterTokenizerFactory extends BaseTokenizerFactory {


  public ArabicLetterTokenizer create(Reader input) {
    return new ArabicLetterTokenizer( input );
  }

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.