Package edu.harvard.wcfia.yoshikoder.document.tokenizer

Examples of edu.harvard.wcfia.yoshikoder.document.tokenizer.TokenizationService


    }
   
    protected Map<YKDocument,Concordance> makeMultipleDocumentConcordance(YKDocument[] docs, Node n, int wsize) throws TokenizationException, IOException {
      //Concordance con = new ConcordanceImpl(wsize);
      TokenizationCache tcache = yoshikoder.getTokenizationCache();
      TokenizationService service = TokenizationService.getTokenizationService();
      Map<YKDocument,Concordance> map = new HashMap<YKDocument,Concordance>();
      for (YKDocument doc : docs) {
        TokenList tl = tcache.getTokenList(doc);
        if (tl == null){
          tl = service.tokenize(doc);
          tcache.putTokenList(doc, tl);
        }
        Concordance c = yoshikoder.getDictionary().getConcordance(tl, n, wsize);
        map.put(doc, c);
      }
View Full Code Here

TOP

Related Classes of edu.harvard.wcfia.yoshikoder.document.tokenizer.TokenizationService

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.