Examples of Tokenizer

  • org.apache.jena.riot.tokens.Tokenizer
  • org.apache.lucene.analysis.Tokenizer
    A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class.

    NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.

    NOTE: Subclasses overriding {@link #incrementToken()} mustcall {@link AttributeSource#clearAttributes()} beforesetting attributes. Subclasses overriding {@link #next(Token)} must call{@link Token#clear()} before setting Token attributes.

  • org.apache.myfaces.trinidadinternal.el.Tokenizer
    converts a EL expression into tokens. @author The Oracle ADF Faces Team
  • org.apache.uima.lucas.indexer.Tokenizer
  • org.crsh.cli.impl.tokenizer.Tokenizer
  • org.eclipse.orion.server.cf.manifest.v2.Tokenizer
  • org.eclipse.osgi.framework.internal.core.Tokenizer
    Simple tokenizer class. Used to parse data.
  • org.exist.storage.analysis.Tokenizer
  • org.geoserver.ows.util.KvpUtils.Tokenizer
  • org.hsqldb.Tokenizer
    Provides the ability to tokenize SQL character sequences. Extensively rewritten and extended in successive versions of HSQLDB. @author Thomas Mueller (Hypersonic SQL Group) @version 1.8.0 @since Hypersonic SQL
  • org.jboss.dna.common.text.TokenStream.Tokenizer
  • org.jboss.forge.shell.command.parser.Tokenizer
    @author Lincoln Baxter, III
  • org.jstripe.tokenizer.Tokenizer
  • org.languagetool.tokenizers.Tokenizer
    Interface for classes that tokenize text into smaller units. @author Daniel Naber
  • org.modeshape.common.text.TokenStream.Tokenizer
  • org.openjena.riot.tokens.Tokenizer
  • org.radargun.utils.Tokenizer
    Tokenizer that allows string delims instead of char delims @author Radim Vansa <rvansa@redhat.com>
  • org.sonatype.maven.polyglot.atom.parsing.Tokenizer
    Taken from the Loop programming language compiler pipeline. @author dhanji@gmail.com (Dhanji R. Prasanna)
  • org.spoofax.jsglr.client.imploder.Tokenizer
  • org.supercsv_voltpatches.tokenizer.Tokenizer
    Reads the CSV file, line by line. If you want the line-reading functionality of this class, but want to define your own implementation of {@link #readColumns(List)}, then consider writing your own Tokenizer by extending AbstractTokenizer. @author Kasper B. Graversen @author James Bassett
  • org.zkoss.selector.lang.Tokenizer
    @author simonpai
  • weka.core.tokenizers.Tokenizer
    A superclass for all tokenizer algorithms. @author FracPete (fracpete at waikato dot ac dot nz) @version $Revision: 1.3 $

  • Examples of org.openjena.riot.tokens.Tokenizer

        }
       
        /** Create a parser for TriG, with default behaviour */
        public static LangTriG createParserTriG(InputStream input, String baseIRI, Sink<Quad> sink)
        {
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerUTF8(input) ;
            return createParserTriG(tokenizer, baseIRI, sink) ;
        }
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

        }

        /** Create a parser for N-Triples, with default behaviour */
        public static LangNTriples createParserNTriples(InputStream input, Sink<Triple> sink)
        {
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerASCII(input) ;
            return createParserNTriples(tokenizer, sink) ;
        }
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

        }
       
        /** Create a parser for NQuads, with default behaviour */
        public static LangNQuads createParserNQuads(InputStream input, Sink<Quad> sink)
        {
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerASCII(input) ;
            return createParserNQuads(tokenizer, sink) ;
        }
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

        }
       
        /** Parse a string to get one Node (the first token in the string) */
        public static Node parse(String string)
        {
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerString(string) ;
            if ( ! tokenizer.hasNext() )
                return null ;
            Token t = tokenizer.next();
            Node n = profile.create(null, t) ;
            if ( tokenizer.hasNext() )
                Log.warn(RiotLib.class, "String has more than one token in it: "+string) ;
            return n ;
        }
    View Full Code Here

    Examples of org.radargun.utils.Tokenizer

       }

       /* Includes special handling for .. operator, but lacks brackets handling as int collection shouldn't contain
        * any sub-collection. */
       private static List<Object> parseIntCollection(String string, Type type) {
          Tokenizer tokenizer = new Tokenizer(string.trim(), new String[] { ",", ".."}, true, false, -1);
          List list = new ArrayList<Object>();

          Integer lastNumber = null;
          boolean generateRange = false;
          boolean firstToken = true;
          while (tokenizer.hasMoreTokens()) {
             String token = tokenizer.nextToken().trim();
             if (token.equals(",")) {
                if (firstToken) throw new IllegalArgumentException("Unexpected ',': " + string);
             } else if (token.equals("..")) {
                if (firstToken) throw new IllegalArgumentException("Unexpected '..': " + string);
                generateRange = true;
    View Full Code Here

    Examples of org.sonatype.maven.polyglot.atom.parsing.Tokenizer

      public Model read(final Reader input, final Map<String, ?> options) throws IOException {
        assert input != null;

        // Parse the token stream from our pom.atom configuration file.
        Project project = new AtomParser((ModelSource)options.get(ModelProcessor.SOURCE), new Tokenizer(IOUtil.toString(input)).tokenize()).parse();
        return project.toMavenModel();
      }
    View Full Code Here

    Examples of org.spoofax.jsglr.client.imploder.Tokenizer

          pt = new ParseTable(tbl, f);
        } catch (InvalidParseTableException e) {
          throw new RuntimeException(e);
        }

        Tokenizer tokenizer = new Tokenizer(" ", " ", new KeywordRecognizer(pt) {});
        Token tok = tokenizer.makeToken(0, IToken.TK_UNKNOWN, true);
        IStrategoTerm term = ATermCommands.makeList("CompilationUnit", tok);
       
        Result r = new Result() {
          public boolean isUpToDate(int h, Environment env) { return false; }
        };
    View Full Code Here

    Examples of org.supercsv_voltpatches.tokenizer.Tokenizer

            final CSVConfig cfg = new CSVConfig();
            cfg.parse(CSVLoader.class.getName(), args);

            config = cfg;
            configuration();
            final Tokenizer tokenizer;
            ICsvListReader listReader = null;
            try {
                if (CSVLoader.standin) {
                    tokenizer = new Tokenizer(new BufferedReader(new InputStreamReader(System.in)), csvPreference,
                            config.strictquotes, config.escape, config.columnsizelimit,
                            config.skip);
                    listReader = new CsvListReader(tokenizer, csvPreference);
                } else {
                    tokenizer = new Tokenizer(new FileReader(config.file), csvPreference,
                            config.strictquotes, config.escape, config.columnsizelimit,
                            config.skip);
                    listReader = new CsvListReader(tokenizer, csvPreference);
                }
            } catch (FileNotFoundException e) {
    View Full Code Here

    Examples of org.zkoss.selector.lang.Tokenizer

       
        //String selector = "div#id.class span.class2 > #id3 ~ intbox";
        //String selector = "[attr$=\"value\"]    +   div:pseudo(2)[attr2*=596]  tab#kerker";
        String selector = ":first-child";
       
        List<Token> tokens = new Tokenizer().tokenize(selector);
        Parser p = new Parser();
        p.setDebugMode(true);
       
        Selector model = p.parse(tokens, selector);
       
    View Full Code Here

    Examples of weka.core.tokenizers.Tokenizer

          String[] tokenizerSpec = Utils.splitOptions(tokenizerString);
          if (tokenizerSpec.length == 0)
      throw new Exception("Invalid tokenizer specification string");
          String tokenizerName = tokenizerSpec[0];
          tokenizerSpec[0] = "";
          Tokenizer tokenizer = (Tokenizer) Class.forName(tokenizerName).newInstance();
          if (tokenizer instanceof OptionHandler)
      ((OptionHandler) tokenizer).setOptions(tokenizerSpec);
          setTokenizer(tokenizer);
        }
      }
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.