Examples of Tokenizer

  • org.apache.jena.riot.tokens.Tokenizer
  • org.apache.lucene.analysis.Tokenizer
    A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class.

    NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.

    NOTE: Subclasses overriding {@link #incrementToken()} mustcall {@link AttributeSource#clearAttributes()} beforesetting attributes. Subclasses overriding {@link #next(Token)} must call{@link Token#clear()} before setting Token attributes.

  • org.apache.myfaces.trinidadinternal.el.Tokenizer
    converts a EL expression into tokens. @author The Oracle ADF Faces Team
  • org.apache.uima.lucas.indexer.Tokenizer
  • org.crsh.cli.impl.tokenizer.Tokenizer
  • org.eclipse.orion.server.cf.manifest.v2.Tokenizer
  • org.eclipse.osgi.framework.internal.core.Tokenizer
    Simple tokenizer class. Used to parse data.
  • org.exist.storage.analysis.Tokenizer
  • org.geoserver.ows.util.KvpUtils.Tokenizer
  • org.hsqldb.Tokenizer
    Provides the ability to tokenize SQL character sequences. Extensively rewritten and extended in successive versions of HSQLDB. @author Thomas Mueller (Hypersonic SQL Group) @version 1.8.0 @since Hypersonic SQL
  • org.jboss.dna.common.text.TokenStream.Tokenizer
  • org.jboss.forge.shell.command.parser.Tokenizer
    @author Lincoln Baxter, III
  • org.jstripe.tokenizer.Tokenizer
  • org.languagetool.tokenizers.Tokenizer
    Interface for classes that tokenize text into smaller units. @author Daniel Naber
  • org.modeshape.common.text.TokenStream.Tokenizer
  • org.openjena.riot.tokens.Tokenizer
  • org.radargun.utils.Tokenizer
    Tokenizer that allows string delims instead of char delims @author Radim Vansa <rvansa@redhat.com>
  • org.sonatype.maven.polyglot.atom.parsing.Tokenizer
    Taken from the Loop programming language compiler pipeline. @author dhanji@gmail.com (Dhanji R. Prasanna)
  • org.spoofax.jsglr.client.imploder.Tokenizer
  • org.supercsv_voltpatches.tokenizer.Tokenizer
    Reads the CSV file, line by line. If you want the line-reading functionality of this class, but want to define your own implementation of {@link #readColumns(List)}, then consider writing your own Tokenizer by extending AbstractTokenizer. @author Kasper B. Graversen @author James Bassett
  • org.zkoss.selector.lang.Tokenizer
    @author simonpai
  • weka.core.tokenizers.Tokenizer
    A superclass for all tokenizer algorithms. @author FracPete (fracpete at waikato dot ac dot nz) @version $Revision: 1.3 $

    • Examples of com.googlecode.psiprobe.tokenizer.Tokenizer

    324
    325
    326
    327
    328
    329
    330
    331
    332
    333
    334
    335
    336
    337
    338
    339
    340
    341
    342
    343
    344
    345
    346
                ByteArrayOutputStream bos = new ByteArrayOutputStream();
                jspRenderer.highlight(name, input, bos, encoding, true);

                ByteArrayInputStream bis = new ByteArrayInputStream(bos.toByteArray());

                Tokenizer tokenizer = new Tokenizer(new InputStreamReader(bis, encoding));
                tokenizer.addSymbol(new TokenizerSymbol("EOL", "\n", null, false, false, true, false));
                tokenizer.addSymbol(new TokenizerSymbol("EOL", "\r\n", null, false, false, true, false));

                //
                // JHighlight adds HTML comment as the first line, so if
                // we number the lines we could end up with a line number and no line
                // to avoid that we just ignore the first line alltogether.
                //
                StringBuffer buffer = new StringBuffer();
                long counter = 0;
                while (tokenizer.hasMore()) {
                    Token tk = tokenizer.nextToken();
                    if ("EOL".equals(tk.getName())) {
                        counter++;
                        buffer.append(tk.getText());
                    } else if (counter > 0) {
                        buffer.append("<span class=\"codeline\">");
    View Full Code Here

    Examples of com.hp.hpl.jena.util.Tokenizer

    675
    676
    677
    678
    679
    680
    681
    682
            /**
             * Constructor
             * @param source the string to be parsed
             */
            Parser(String source) {
                stream = new Tokenizer(source, "()[], \t\n\r", "'\"", true);
                lookahead = null;
            }
    View Full Code Here

    Examples of com.intellij.spellchecker.tokenizer.Tokenizer

    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
      @NotNull
      @Override
      public Tokenizer getTokenizer(PsiElement element) {
        if (element instanceof CfmlStringLiteralExpression) {
          return new Tokenizer() {
            @Override
            public void tokenize(@NotNull final PsiElement element, TokenConsumer consumer) {
              consumer.consumeToken(element, new TextSplitter() {
                @Override
                public void split(@Nullable String text, @NotNull TextRange range, Consumer<TextRange> consumer) {
    View Full Code Here

    Examples of com.metaweb.lessen.tokenizers.Tokenizer

    528
    529
    530
    531
    532
    533
    534
    535
    536
    537
    538
            URL url = getResource(path);
           
            Map<String, String> variables = new HashMap<String, String>();
            variables.put("module", _name);
           
            Tokenizer tokenizer = Utilities.openLess(url, variables);
            tokenizer = new CondensingTokenizer(tokenizer, false);
            tokenizer = new IndentingTokenizer(tokenizer);
           
            return sendLessenTokenStream(request, response, tokenizer, encoding, "text/css",false);
        }
    View Full Code Here

    Examples of com.sun.enterprise.admin.util.Tokenizer

    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
        throws com.sun.enterprise.admin.util.TokenizerException
      {
        final String  delimiters    = "" + ARRAY_ELEMENT_SEPARATOR;
        final String  escapableChars  = "" + ARRAY_ELEMENT_SEPARATOR + ESCAPE_CHAR;
       
        final Tokenizer  tok  = new TokenizerImpl( s, delimiters, false, ESCAPE_CHAR, escapableChars );
       
        final String []  values  = tok.getTokens();
       
        return( values );
      }
    View Full Code Here

    Examples of com.sun.enterprise.module.common_impl.Tokenizer

    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
        MavenModuleDefinition(MavenProjectRepository repository, File location) throws IOException {
            super(location);

            try {
                String classpath = mainAttributes.getValue(ManifestConstants.CLASS_PATH_ID);
                for( String id : new Tokenizer(classpath," ")) {
                    File jar = repository.resolveArtifact(id);
                    classPath.add(jar.toURI());
                }
            } catch (IOException e) {
                throw new IOException2("Failed to process "+ManifestConstants.CLASS_PATH_ID+" for "+location,e);
    View Full Code Here

    Examples of com.sun.speech.freetts.Tokenizer

    297
    298
    299
    300
    301
    302
    303
    304
    305
    306
    307
    308
         * Gets a tokenizer for this voice
         *
         * @return the tokenizer
         */
        public Tokenizer getTokenizer() {
      Tokenizer tokenizer = new com.sun.speech.freetts.en.TokenizerImpl();
      tokenizer.setWhitespaceSymbols(USEnglish.WHITESPACE_SYMBOLS);
      tokenizer.setSingleCharSymbols(USEnglish.SINGLE_CHAR_SYMBOLS);
      tokenizer.setPrepunctuationSymbols(USEnglish.PREPUNCTUATION_SYMBOLS);
      tokenizer.setPostpunctuationSymbols(USEnglish.PUNCTUATION_SYMBOLS);
      return tokenizer;
        }
    View Full Code Here

    Examples of edu.buffalo.cse.ir.wikiindexer.tokenizer.Tokenizer

    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
       
        idoc.docId = doc1.getId();
       
       
        TokenStream author = new TokenStream(doc1.getAuthor());
        Tokenizer t_author = tknizerMap1.get(INDEXFIELD.AUTHOR);
        t_author.tokenize(author);
        //System.out.println("=======" +author.getAllTokens());
       
       
        TokenStream categories = new TokenStream(doc1.getCategories().toString());
        Tokenizer t_categories = tknizerMap1.get(INDEXFIELD.CATEGORY);
        t_categories.tokenize(categories);
        //System.out.println("=======" +categories.getAllTokens());
       
        TokenStream links = new TokenStream(doc1.getLinks().toString());
        Tokenizer t_links = tknizerMap1.get(INDEXFIELD.LINK);
        t_links.tokenize(links);
        //System.out.println("=======" +links.getAllTokens());
       
        TokenStream term = new TokenStream(doc1.getSections().get(0).getText());
        for(int i=1;i<doc1.getSections().size();i++)
        {
          term.append(doc1.getSections().get(i).getText());
        }
        Tokenizer t_term = tknizerMap1.get(INDEXFIELD.TERM);
        t_term.tokenize(term);
        //System.out.println("=======" +term.getAllTokens());
       
       
          idoc.addField(INDEXFIELD.AUTHOR, author);
          idoc.addField(INDEXFIELD.CATEGORY, categories);
    View Full Code Here

    Examples of edu.harvard.wcfia.yoshikoder.document.tokenizer.Tokenizer

    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
            YKDocument d1 = YKDocumentFactory.createDummyDocument("D1", "Mary had a little lamb.  Mary had some more", "UTF-8");
            YKDocument d2 = YKDocumentFactory.createDummyDocument("D1", "Jackie had a little beef.  Jackie whined some more", "UTF-8");
            DocumentList dl = new DocumentListImpl();
            dl.add(d1);
            dl.add(d2);
            Tokenizer tok = new BITokenizerImpl();
            WordFrequencyMap wd1 = new WordFrequencyMap(tok.getTokens(d1.getText()));
            WordFrequencyMap wd2 = new WordFrequencyMap(tok.getTokens(d2.getText()));
            UnifiedDocumentFrequencyReport rep = new UnifiedDocumentFrequencyReport("title", "desc",
                   dl, new WordFrequencyMap[]{wd1, wd2});
            JTable table = new JTable(rep);
            JOptionPane.showMessageDialog(null, new JScrollPane(table));
        }
    View Full Code Here

    Examples of edu.isi.karma.cleaning.Tokenizer

    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
        fnames = new Vector<String>();
      }

      public Vector<TNode> tokenizer(String Org) {
        CharStream cs = new ANTLRStringStream(Org);
        Tokenizer tk = new Tokenizer(cs);
        Token t;
        t = tk.nextToken();
        Vector<TNode> x = new Vector<TNode>();
        while (t.getType() != -1) {
          int mytype = -1;
          if (t.getType() == 15) {
            mytype = TNode.UWRDTYP;
          } else if (t.getType() == 4) {
            mytype = TNode.BNKTYP;
          } else if (t.getType() == 10) {
            mytype = TNode.NUMTYP;
          } else if (t.getType() == 12) {
            mytype = TNode.SYBSTYP;
          } else if (t.getType() == 9) {
            mytype = TNode.LWRDTYP;
          }
          TNode tx = new TNode(mytype, t.getText());
          x.add(tx);
          t = tk.nextToken();
        }
        return x;
      }
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.