Examples of TokenizerME

opennlp.tools.tokenize.TokenizerME
is.upenn.edu/~jcreynar>.
This tokenizer needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The {@link TokenizerModel} class encapsulates the model and providesmethods to create it from the binary representation.
A tokenizer instance is not thread safe. For each thread one tokenizer must be instantiated which can share one TokenizerModel instance to safe memory.
To train a new model { {@link #train(String,ObjectStream,boolean,TrainingParameters)} methodcan be used.
Sample usage:
InputStream modelIn; ... TokenizerModel model = TokenizerModel(modelIn); Tokenizer tokenizer = new TokenizerME(model); String tokens[] = tokenizer.tokenize("A sentence to be tokenized."); @see Tokenizer @see TokenizerModel @see TokenSample

Examples of opennlp.tools.tokenize.TokenizerME

     * 
     * @throws IOException 
     */
    public ApacheExtractor() throws IOException {
        nameFinder = new NameFinderME(new TokenNameFinderModel(ApacheExtractor.class.getResourceAsStream(pathToNERModel)));
        tokenizer = new TokenizerME(new TokenizerModel(ApacheExtractor.class.getResourceAsStream(pathToTokenizerModel)));
        sentenceDetector = new SentenceDetectorME(new SentenceModel(ApacheExtractor.class.getResourceAsStream(pathToSentenceDetectorModel)));
    }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

            tokenizerModel = new TokenizerModel(tokenizerModelInputStream);
        } finally {
            tokenizerModelInputStream.close();
        }


        return new TokenizerME(tokenizerModel);
    }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

            tokenizerModel = new TokenizerModel(tokenizerModelInputStream);
        } finally {
            tokenizerModelInputStream.close();
        }


        return new TokenizerME(tokenizerModel);
    }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

      model = modelResource.getModel();
    } catch (ResourceAccessException e) {
      throw new ResourceInitializationException(e);
    }


    tokenizer = new TokenizerME(model);
  }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

      model = modelResource.getModel();
    } catch (ResourceAccessException e) {
      throw new ResourceInitializationException(e);
    }


    tokenizer = new TokenizerME(model);
  }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

   * 
   * @param text
   */
  public SentencesToTree(String text, TokenizerModel model){
    /* Configure the tokenizer with preloaded model */
    Tokenizer tokenizer = new TokenizerME(model);
    /* tokens has an array of strings, where each string is a token */
    String s = spaces(tokenizer.tokenize(text));
    this.text = this.upperCase(s);
  }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

  public ObjectStream<NameSample> create(String[] args) {


    Parameters params = ArgumentParser.parse(args, Parameters.class);


    TokenizerModel tokenizerModel = new TokenizerModelLoader().load(params.getTokenizerModel());
    Tokenizer tokenizer = new TokenizerME(tokenizerModel);


    ObjectStream<String> mucDocStream = new FileToStringSampleStream(
        new DirectorySampleStream(params.getData(), new FileFilter() {


          public boolean accept(File file) {

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

    
    ParserModel parserModel = new ParserModelLoader().load(params.getParserModel());
    Parser parser =  ParserFactory.create(parserModel);
    
    TokenizerModel tokenizerModel = new TokenizerModelLoader().load(params.getTokenizerModel());
    Tokenizer tokenizer = new TokenizerME(tokenizerModel);
    
    ObjectStream<String> mucDocStream = new FileToStringSampleStream(
        new DirectorySampleStream(params.getData(), new FileFilter() {
          
          public boolean accept(File file) {

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

  }


  @Override
  public Resource init() throws ResourceInstantiationException {
    //logger.info("The string of Tokenizer file is: "+model);
    tokenizer = new TokenizerME(
        getModel(model));
    
    logger.warn("OpenNLP Tokenizer initialized!");//System.out.println("OpenNLP Tokenizer initialized!");
    
    return this;

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME

  public void setTokenizer(FileSystem fs, Path p){
    try {
      FSDataInputStream in = fs.open(p);
      TokenizerModel model;
      model = new TokenizerModel(in);
      tokenizer = new TokenizerME(model);
    }
    catch (IOException e) {
      e.printStackTrace();
    }
  }

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.