Examples of de.susebox.jtopas.StandardTokenizer

de.susebox.jtopas.StandardTokenizer

This is the mainstream {@link Tokenizer}. It implements the {@link Tokenizer}interface in a straightforward approach without too specialized parse optimizations.

Beside the {@link Tokenizer} interface, the class StandardTokenizerprovides some basic features for cascading (nested) tokenizers. Consider the usual HTML pages found today in the WWW. Most of them are a mixture of regular HTML, cascading style sheets (CSS) and embedded JavaScript. These different languages use different syntaxes, so one needs varous tokenizers on the same input stream.

This {@link Tokenizer} implementation is not synchronized. Take care when usingwith multible threads.

@see Tokenizer @see TokenizerProperties @author Heiko Blau

   */
  private long tokenizeFile(String message, File file, boolean useReaderSource) throws Throwable {
    System.out.println(message);


    TokenizerSource   source    = null;
    StandardTokenizer tokenizer = new StandardTokenizer(_properties);
    
    try {
      if (useReaderSource) {
        source = new ReaderSource(file);
      } else {
        source = new CharArraySource(readFile(file));
      }
      tokenizer.setSource(source);
      return tokenize(tokenizer);
    } finally {
      if (useReaderSource && source != null) {
        ((ReaderSource)source).close();
      }
      tokenizer.close();
    }
  }

Examples of de.susebox.jtopas.StandardTokenizer

Related Classes of de.susebox.jtopas.StandardTokenizer