Examples of LimitTokenCountFilter

org.apache.lucene.analysis.LimitTokenCountFilter

This TokenFilter limits the number of tokens while indexing. It is a replacement for the maximum field length setting inside {@link org.apache.lucene.index.IndexWriter}.

org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter

This TokenFilter limits the number of tokens while indexing. It is a replacement for the maximum field length setting inside {@link org.apache.lucene.index.IndexWriter}.

By default, this filter ignores any tokens in the wrapped {@code TokenStream}once the limit has been reached, which can result in {@code reset()} being called prior to {@code incrementToken()} returning {@code false}. For most {@code TokenStream} implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a {@code TokenStream} which requires that the full stream of tokens be exhausted in order to function properly, use the {@link #LimitTokenCountFilter(TokenStream,int,boolean) consumeAllTokens} option.

    super.init( args );
    maxTokenCount = Integer.parseInt( args.get( "maxTokenCount" ) );
  }


  public TokenStream create(TokenStream input) {
    return new LimitTokenCountFilter( input, maxTokenCount );
  }

    consumeAllTokens = getBoolean(CONSUME_ALL_TOKENS_KEY, false);
  }


  @Override
  public TokenStream create(TokenStream input) {
    return new LimitTokenCountFilter(input, maxTokenCount, consumeAllTokens);
  }

        this.consumeAllTokens = settings.getAsBoolean("consume_all_tokens", DEFAULT_CONSUME_ALL_TOKENS);
    }


    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new LimitTokenCountFilter(tokenStream, maxTokenCount, consumeAllTokens);
    }

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of LimitTokenCountFilter

Examples of org.apache.lucene.analysis.LimitTokenCountFilter

Examples of org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter

Examples of org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter