Examples of TokenCounterPersistence

com.flaptor.hounder.classifier.util.TokenCounterPersistence
In previous versions, we added the data to a Map(token->TokenCounter) in memory. However such aproach consumes too much RAM and was discarded. In this version we add the data to a file that is lately transformed into a Map backed by a file. This class implements a file where partial data is saved, for late sort and add to the map @author rafa @see BayesCalculator#computeProbabilities() @see BayesCalculator#addData(java.util.Map,boolean,String)

Examples of com.flaptor.hounder.classifier.util.TokenCounterPersistence

        this.categoryName = categoryName;
        dataDir = dir;
        myProbabilities = PersistenceManager.readProbabilitiesFromFile(dataDir,
                categoryName+".my.probabilities");
        try {
            tokenCountersFile= new TokenCounterPersistence().newRecordWriter( 
                    new File(CAT_TOK_FILE_UNSORTED));
        } catch (FileNotFoundException e) {
            e.printStackTrace();
            throw (e);
        }

View Full Code Here

Examples of com.flaptor.hounder.classifier.util.TokenCounterPersistence

         */
        TokenCounterPersistence.TCRecord tcr;
        if (belongsToCategory) {
            categoryDocumentCount++;
            for (String token : tokens) {
                tcr= new TokenCounterPersistence().newRecord(token, 
                        documentTokenCount.get(token)[0], 0);
                tokenCountersFile.writeRecord(tcr);
            }
        } else {
            nonCategoryDocumentCount++;
            for (String token : tokens) {
                tcr= new TokenCounterPersistence().newRecord(token, 0, 
                        documentTokenCount.get(token)[0]);
                tokenCountersFile.writeRecord(tcr);
            }            
        }
    }

View Full Code Here

Examples of com.flaptor.hounder.classifier.util.TokenCounterPersistence

        tokenCountersFile.close();        
        logger.info("Closing categoryTokenCount.... done");
        logger.info("Sorting categoryTokenCount....");
        File beforeSort= new File(CAT_TOK_FILE_UNSORTED);
        File afterSort= new File(CAT_TOK_FILE_SORTED);
        TokenCounterPersistence tcp= new TokenCounterPersistence();
        MergeSort.sort(beforeSort, afterSort, tcp, null);
        logger.info("Sorting categoryTokenCount.... done");


        logger.info("Folding categoryTokenCount.... ");
        // Now saves all this data to a FileCache        
        RecordReader tcrr= tcp.newRecordReader(afterSort);                
        TokenCounter catTc= new TokenCounter();
        TokenCounter nonCatTc= new TokenCounter();
        TokenCounterPersistence.TCRecord tcr= (TokenCounterPersistence.TCRecord) tcrr.readRecord();
        if (null==tcrr) return;
        catTc.update(tcr.getCatVal());

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.