Examples of Tokenizer

  • org.apache.jena.riot.tokens.Tokenizer
  • org.apache.lucene.analysis.Tokenizer
    A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class.

    NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.

    NOTE: Subclasses overriding {@link #incrementToken()} mustcall {@link AttributeSource#clearAttributes()} beforesetting attributes. Subclasses overriding {@link #next(Token)} must call{@link Token#clear()} before setting Token attributes.

  • org.apache.myfaces.trinidadinternal.el.Tokenizer
    converts a EL expression into tokens. @author The Oracle ADF Faces Team
  • org.apache.uima.lucas.indexer.Tokenizer
  • org.crsh.cli.impl.tokenizer.Tokenizer
  • org.eclipse.orion.server.cf.manifest.v2.Tokenizer
  • org.eclipse.osgi.framework.internal.core.Tokenizer
    Simple tokenizer class. Used to parse data.
  • org.exist.storage.analysis.Tokenizer
  • org.geoserver.ows.util.KvpUtils.Tokenizer
  • org.hsqldb.Tokenizer
    Provides the ability to tokenize SQL character sequences. Extensively rewritten and extended in successive versions of HSQLDB. @author Thomas Mueller (Hypersonic SQL Group) @version 1.8.0 @since Hypersonic SQL
  • org.jboss.dna.common.text.TokenStream.Tokenizer
  • org.jboss.forge.shell.command.parser.Tokenizer
    @author Lincoln Baxter, III
  • org.jstripe.tokenizer.Tokenizer
  • org.languagetool.tokenizers.Tokenizer
    Interface for classes that tokenize text into smaller units. @author Daniel Naber
  • org.modeshape.common.text.TokenStream.Tokenizer
  • org.openjena.riot.tokens.Tokenizer
  • org.radargun.utils.Tokenizer
    Tokenizer that allows string delims instead of char delims @author Radim Vansa <rvansa@redhat.com>
  • org.sonatype.maven.polyglot.atom.parsing.Tokenizer
    Taken from the Loop programming language compiler pipeline. @author dhanji@gmail.com (Dhanji R. Prasanna)
  • org.spoofax.jsglr.client.imploder.Tokenizer
  • org.supercsv_voltpatches.tokenizer.Tokenizer
    Reads the CSV file, line by line. If you want the line-reading functionality of this class, but want to define your own implementation of {@link #readColumns(List)}, then consider writing your own Tokenizer by extending AbstractTokenizer. @author Kasper B. Graversen @author James Bassett
  • org.zkoss.selector.lang.Tokenizer
    @author simonpai
  • weka.core.tokenizers.Tokenizer
    A superclass for all tokenizer algorithms. @author FracPete (fracpete at waikato dot ac dot nz) @version $Revision: 1.3 $

  • Examples of org.apache.lucene.analysis.Tokenizer

          final boolean ignoreCase = random.nextBoolean();
         
          final Analyzer analyzer = new Analyzer() {
            @Override
            protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
              Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.SIMPLE, true);
              TokenStream syns = new SynonymFilter(tokenizer, map, ignoreCase);
              TokenStream graph = new MockGraphTokenFilter(random(), syns);
              return new TokenStreamComponents(tokenizer, graph);
            }
          };
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

          final boolean ignoreCase = random.nextBoolean();
         
          final Analyzer analyzer = new Analyzer() {
            @Override
            protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
              Tokenizer tokenizer = new KeywordTokenizer(reader);
              return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, ignoreCase));
            }
          };

          checkAnalysisConsistency(random, analyzer, random.nextBoolean(), "");
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

          final boolean ignoreCase = random.nextBoolean();
         
          final Analyzer analyzer = new Analyzer() {
            @Override
            protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
              Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.SIMPLE, true);
              return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, ignoreCase));
            }
          };

          checkRandomData(random, analyzer, 100, 1024);
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        final SynonymMap map = parser.build();
         
        Analyzer analyzer = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, true);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };
       
        // where did my pot go?!
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

          final int min = _TestUtil.nextInt(random(), 2, 10);
          final int max = _TestUtil.nextInt(random(), min, 20);
          Analyzer a = new Analyzer() {
            @Override
            protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
              Tokenizer tokenizer = new NGramTokenizer(TEST_VERSION_CURRENT, reader, min, max);
              return new TokenStreamComponents(tokenizer, tokenizer);
            }   
          };
          checkRandomData(random(), a, 200*RANDOM_MULTIPLIER, 20);
          checkRandomData(random(), a, 10*RANDOM_MULTIPLIER, 1027);
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        add("x c", "xc", keepOrig);
        final SynonymMap map = b.build();
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };

        checkOneTerm(a, "$", "$");
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        add("a b", "ab", keepOrig);
        final SynonymMap map = b.build();
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };

        assertAnalyzesTo(a, "a b",
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        add("a b", "ab", keepOrig);
        final SynonymMap map = b.build();
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };

        assertAnalyzesTo(a, "a b",
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        add("zoo", "zoo", keepOrig);
        final SynonymMap map = b.build();
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };
       
        assertAnalyzesTo(a, "zoo zoo $ zoo",
    View Full Code Here

    Examples of org.apache.lucene.analysis.Tokenizer

        add("zoo", "zoo zoo", keepOrig);
        final SynonymMap map = b.build();
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
            return new TokenStreamComponents(tokenizer, new SynonymFilter(tokenizer, map, true));
          }
        };

        // verify("zoo zoo $ zoo", "zoo/zoo zoo/zoo/zoo $/zoo zoo/zoo zoo");
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.