Examples of Tokenizer

  • org.apache.jena.riot.tokens.Tokenizer
  • org.apache.lucene.analysis.Tokenizer
    A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class.

    NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.

    NOTE: Subclasses overriding {@link #incrementToken()} mustcall {@link AttributeSource#clearAttributes()} beforesetting attributes. Subclasses overriding {@link #next(Token)} must call{@link Token#clear()} before setting Token attributes.

  • org.apache.myfaces.trinidadinternal.el.Tokenizer
    converts a EL expression into tokens. @author The Oracle ADF Faces Team
  • org.apache.uima.lucas.indexer.Tokenizer
  • org.crsh.cli.impl.tokenizer.Tokenizer
  • org.eclipse.orion.server.cf.manifest.v2.Tokenizer
  • org.eclipse.osgi.framework.internal.core.Tokenizer
    Simple tokenizer class. Used to parse data.
  • org.exist.storage.analysis.Tokenizer
  • org.geoserver.ows.util.KvpUtils.Tokenizer
  • org.hsqldb.Tokenizer
    Provides the ability to tokenize SQL character sequences. Extensively rewritten and extended in successive versions of HSQLDB. @author Thomas Mueller (Hypersonic SQL Group) @version 1.8.0 @since Hypersonic SQL
  • org.jboss.dna.common.text.TokenStream.Tokenizer
  • org.jboss.forge.shell.command.parser.Tokenizer
    @author Lincoln Baxter, III
  • org.jstripe.tokenizer.Tokenizer
  • org.languagetool.tokenizers.Tokenizer
    Interface for classes that tokenize text into smaller units. @author Daniel Naber
  • org.modeshape.common.text.TokenStream.Tokenizer
  • org.openjena.riot.tokens.Tokenizer
  • org.radargun.utils.Tokenizer
    Tokenizer that allows string delims instead of char delims @author Radim Vansa <rvansa@redhat.com>
  • org.sonatype.maven.polyglot.atom.parsing.Tokenizer
    Taken from the Loop programming language compiler pipeline. @author dhanji@gmail.com (Dhanji R. Prasanna)
  • org.spoofax.jsglr.client.imploder.Tokenizer
  • org.supercsv_voltpatches.tokenizer.Tokenizer
    Reads the CSV file, line by line. If you want the line-reading functionality of this class, but want to define your own implementation of {@link #readColumns(List)}, then consider writing your own Tokenizer by extending AbstractTokenizer. @author Kasper B. Graversen @author James Bassett
  • org.zkoss.selector.lang.Tokenizer
    @author simonpai
  • weka.core.tokenizers.Tokenizer
    A superclass for all tokenizer algorithms. @author FracPete (fracpete at waikato dot ac dot nz) @version $Revision: 1.3 $

  • Examples of org.geoserver.ows.util.KvpUtils.Tokenizer

        private void checkTypeAxisRange(final RangeSubsetType rangeSubset, Object axis, String axisName) {
            if (axis instanceof String) {
                String bands = (String) axis;
                if (bands != null) {
                    if (bands.contains("/")) {
                        List<String> unparsed = KvpUtils.readFlat(bands, new Tokenizer("/"));

                        IntervalType interval = Wcs10Factory.eINSTANCE.createIntervalType();
                        TypedLiteralType min = Wcs10Factory.eINSTANCE.createTypedLiteralType();
                        TypedLiteralType max = Wcs10Factory.eINSTANCE.createTypedLiteralType();
                        TypedLiteralType res = Wcs10Factory.eINSTANCE.createTypedLiteralType();
    View Full Code Here

    Examples of org.hsqldb.Tokenizer

        public RowInputTextLog() {

            super(new byte[0]);

            tokenizer = new Tokenizer();
        }
    View Full Code Here

    Examples of org.jboss.dna.common.text.TokenStream.Tokenizer

         * @throws ParsingException if there is an error parsing the supplied string
         * @throws IllegalArgumentException if the expression is null
         */
        public Term parse( String fullTextSearchExpression ) {
            CheckArg.isNotNull(fullTextSearchExpression, "fullTextSearchExpression");
            Tokenizer tokenizer = TokenStream.basicTokenizer(false);
            TokenStream stream = new TokenStream(fullTextSearchExpression, tokenizer, false);
            return parse(stream.start());
        }
    View Full Code Here

    Examples of org.jboss.forge.shell.command.parser.Tokenizer

       {
          this.buffer = initialBuffer;
          this.lastBuffer = lastBuffer;
          this.index = initialIndex;
          this.originalIndex = initialIndex;
          this.tokens = new Tokenizer().tokenize(initialBuffer);
          this.tokenComplete = buffer.matches("^.*\\s+$");
       }
    View Full Code Here

    Examples of org.jstripe.tokenizer.Tokenizer

        public LineReader(InputStream stream, boolean backwards) {
            this.readBackwards = backwards;

            this.streamReader = new InputStreamReader(stream);
            tokenizer = new Tokenizer(streamReader);
            tokenizer.addSymbol(new TokenizerSymbol(LINE_SEPARATOR, "\n", null, false, false, true, false));
            tokenizer.addSymbol(new TokenizerSymbol(LINE_SEPARATOR, "\r", null, false, false, true, false));

            if (backwards) {
                tokenizer.addSymbol(new TokenizerSymbol(LINE_SEPARATOR, "\n\r", null, false, false, true, false));
    View Full Code Here

    Examples of org.languagetool.tokenizers.Tokenizer

        } else {
          System.out.println("Checking " + file.getAbsolutePath());
          String text = StringTools.readFile(new FileInputStream(file.getAbsolutePath()));
          text = textFilter.filter(text);
          if (CHECK_BY_SENTENCE) {
            final Tokenizer sentenceTokenizer = langTool.getLanguage().getSentenceTokenizer();
            final List<String> sentences = sentenceTokenizer.tokenize(text);
            for (String sentence : sentences) {
              Tools.checkText(sentence, langTool, false, 1000);
            }
          } else {
            Tools.checkText(text, langTool);
    View Full Code Here

    Examples of org.modeshape.common.text.TokenStream.Tokenizer

         *
         * @param content the content
         * @throws ParsingException if there is a problem parsing the content
         */
        protected void parse( String content ) {
            Tokenizer tokenizer = new CndTokenizer(false, true);
            TokenStream tokens = new TokenStream(content, tokenizer, false);
            tokens.start();
            while (tokens.hasNext()) {
                // Keep reading while we can recognize one of the two types of statements ...
                if (tokens.matches("<", ANY_VALUE, "=", ANY_VALUE, ">")) {
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

            {
                if ( baseIRI != null )
                    baseIRI = IRIResolver.resolveString(baseIRI) ;
                return LangRDFXML.create(input, baseIRI, baseIRI, ErrorHandlerFactory.getDefaultErrorHandler(), sink) ;
            }
            Tokenizer tokenizer = ( lang == RDFJSON ) ?
                new TokenizerJSON(PeekReader.makeUTF8(input)) :
                    TokenizerFactory.makeTokenizerUTF8(input) ;
            return createParserTriples(tokenizer, lang, baseIRI ,sink) ;
        }
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

            if ( lang.isTriples() )
            {
                SinkExtendTriplesToQuads converter = new SinkExtendTriplesToQuads(sink) ;
                return createParserTriples(input, lang, baseIRI, converter) ;
            }
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerUTF8(input) ;
            return createParserQuads(tokenizer, lang, baseIRI, sink) ;
        }
    View Full Code Here

    Examples of org.openjena.riot.tokens.Tokenizer

        }
       
        /** Create a parser for Turtle, with default behaviour */
        public static LangTurtle createParserTurtle(InputStream input, String baseIRI, Sink<Triple> sink)
        {
            Tokenizer tokenizer = TokenizerFactory.makeTokenizerUTF8(input) ;
            return createParserTurtle(tokenizer, baseIRI, sink) ;
        }
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.