Package edu.harvard.wcfia.yoshikoder.document.tokenizer

Examples of edu.harvard.wcfia.yoshikoder.document.tokenizer.Tokenizer


        YKDocument d1 = YKDocumentFactory.createDummyDocument("D1", "Mary had a little lamb.  Mary had some more", "UTF-8");
        YKDocument d2 = YKDocumentFactory.createDummyDocument("D1", "Jackie had a little beef.  Jackie whined some more", "UTF-8");
        DocumentList dl = new DocumentListImpl();
        dl.add(d1);
        dl.add(d2);
        Tokenizer tok = new BITokenizerImpl();
        WordFrequencyMap wd1 = new WordFrequencyMap(tok.getTokens(d1.getText()));
        WordFrequencyMap wd2 = new WordFrequencyMap(tok.getTokens(d2.getText()));
        UnifiedDocumentFrequencyReport rep = new UnifiedDocumentFrequencyReport("title", "desc",
               dl, new WordFrequencyMap[]{wd1, wd2});
        JTable table = new JTable(rep);
        JOptionPane.showMessageDialog(null, new JScrollPane(table));
    }
View Full Code Here

TOP

Related Classes of edu.harvard.wcfia.yoshikoder.document.tokenizer.Tokenizer

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.