Examples of TwoPassDataIndexer

opennlp.maxent.TwoPassDataIndexer
opennlp.model.TwoPassDataIndexer
Collecting event and context counts by making two passes over the events. The first pass determines which contexts will be used by the model, and the second pass creates the events in memory containing only the contexts which will be used. This greatly reduces the amount of memory required for storing the events. During the first pass a temporary event file is created which is read during the second pass.
opennlp.tools.ml.model.TwoPassDataIndexer
Collecting event and context counts by making two passes over the events. The first pass determines which contexts will be used by the model, and the second pass creates the events in memory containing only the contexts which will be used. This greatly reduces the amount of memory required for storing the events. During the first pass a temporary event file is created which is read during the second pass.

Examples of opennlp.maxent.TwoPassDataIndexer

      List<Event> trainData = events.subList(0, events.size()/2);
      //trainData = new FeatureSelector().selectFeatures(trainData, 200.0);
      List<Event> testData = events.subList(events.size()/2, events.size());
      
      if(trainData.size() == 1) trainData.add(trainData.get(0));
      DataIndexer di = new TwoPassDataIndexer(new EventCollectorAsStream(new SimpleEventCollector(trainData)), 1);
      GISModel gm = GIS.trainModel(100, di);
      
      ClassificationEvaluator ce = new ClassificationEvaluator();
      
      for(Event event : testData) {

View Full Code Here

Examples of opennlp.maxent.TwoPassDataIndexer

          events.add(ev);
        }
      }
      if(events.size() < 2) continue;
      DataIndexer di = null;
      di = new TwoPassDataIndexer(new EventCollectorAsStream(new SimpleEventCollector(events)), 1);
      if(Oscar3Props.getInstance().verbose) System.out.println(di);
      GISModel gm = GIS.trainModel(100, di);
      classifiers.put(type, gm);
    }
  }

View Full Code Here

Examples of opennlp.maxent.TwoPassDataIndexer

    
    List<Event> trainData = events.subList(0, events.size()/2);
    List<Event> testData = events.subList(events.size()/2, events.size());
    
    if(trainData.size() == 1) trainData.add(trainData.get(0));
    DataIndexer di = new TwoPassDataIndexer(new EventCollectorAsStream(new SimpleEventCollector(trainData)), 1);
    GISModel gm = GIS.trainModel(100, di);
    
    ClassificationEvaluator ce = new ClassificationEvaluator();
    
    for(Event event : testData) {

View Full Code Here

Examples of opennlp.maxent.TwoPassDataIndexer

        trainEvents.add(new Event(be.getClassLabel(), be.getFeatures().getSet().toArray(new String[0])));
      }
      for(BagEvent be : testBagEvents) {
        testEvents.add(new Event(be.getClassLabel(), be.getFeatures().getSet().toArray(new String[0])));
      }
      DataIndexer di = new TwoPassDataIndexer(new EventCollectorAsStream(new SimpleEventCollector(trainEvents)), 1);
      GISModel gm = GIS.trainModel(100, di);
      
      //ClassificationEvaluator ce = new ClassificationEvaluator();
      
      for(Event event : testEvents) {

View Full Code Here

Examples of opennlp.maxent.TwoPassDataIndexer

    for(int i=0;i<numDocs/2;i++) {
      events.add(docToEvent(ir, i, c));
    }
    
    DataIndexer di = null;
    di = new TwoPassDataIndexer(new EventCollectorAsStream(new SimpleEventCollector(events)), 3);
    GISModel gm = GIS.trainModel(100, di);
  
    Map<String,Double> byProb = new HashMap<String,Double>();
    Map<String,String> res = new HashMap<String,String>();
    int mp = 0;

View Full Code Here

Examples of opennlp.model.TwoPassDataIndexer

    Parser parser = new ChunkParser(chunker, tagger);
    AnswerTypeContextGenerator actg = new AnswerTypeContextGenerator(new File(wordnetDir));
    //<start id="atc.train"/>
    AnswerTypeEventStream es = new AnswerTypeEventStream(trainFile,
            actg, parser);
    GISModel model = GIS.trainModel(100, new TwoPassDataIndexer(es, 3));//<co id="atc.train.do"/>
    new DoccatModel("en", model).serialize(new FileOutputStream(outFile));
    /*
    <calloutlist>
        <callout arearefs="atc.train.do"><para>Using the event stream, which feeds us training examples, do the actual training using OpenNLP's Maxent classifier.</para></callout>
    </calloutlist>

View Full Code Here

Examples of opennlp.model.TwoPassDataIndexer

    return train(languageCode, parseSamples, rules, params);
  }
  
  @Deprecated
  public static AbstractModel train(opennlp.model.EventStream es, int iterations, int cut) throws java.io.IOException {
    return opennlp.maxent.GIS.trainModel(iterations, new TwoPassDataIndexer(es, cut));
  }

View Full Code Here

Examples of opennlp.model.TwoPassDataIndexer

   * @deprecated Please do not use anymore, use the ObjectStream train methods instead! This method
   * will be removed soon.
   */
  @Deprecated
  public static AbstractModel train(opennlp.model.EventStream es, int iterations, int cut) throws java.io.IOException {
    return opennlp.maxent.GIS.trainModel(iterations, new TwoPassDataIndexer(es, cut));
  }

View Full Code Here

Examples of opennlp.model.TwoPassDataIndexer


  @Test
  public void testPerceptronOnPrepAttachData() throws IOException {
    AbstractModel model = 
        new PerceptronTrainer().trainModel(400, 
        new TwoPassDataIndexer(createTrainingStream(), 1, false), 1);


    testModel(model, 0.7650408516959644);
  }

View Full Code Here

Examples of opennlp.model.TwoPassDataIndexer


  @Test
  public void testMaxentOnPrepAttachData() throws IOException {
    AbstractModel model = 
        new GISTrainer(true).trainModel(100, 
        new TwoPassDataIndexer(createTrainingStream(), 1), 1);


    testModel(model, 0.7997028967566229);
  }

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.