Examples of TeraValidate

org.apache.hadoop.examples.terasort.TeraValidate
Generate 1 mapper per a file that checks to make sure the keys are sorted within each file. The mapper also generates "$file:begin", first key and "$file:end", last key. The reduce verifies that all of the start/end items are in order. Any output from the reduce is problem report.
To run the program: bin/hadoop jar hadoop-*-examples.jar teravalidate out-dir report-dir
If there is any output, something is wrong and the output of the reduce will have the problem report.
org.apache.tez.mapreduce.examples.terasort.TeraValidate
Generate 1 mapper per a file that checks to make sure the keys are sorted within each file. The mapper also generates "$file:begin", first key and "$file:end", last key. The reduce verifies that all of the start/end items are in order. Any output from the reduce is problem report.
To run the program: bin/hadoop jar hadoop-*-examples.jar teravalidate out-dir report-dir
If there is any output, something is wrong and the output of the reduce will have the problem report.

Examples of org.apache.hadoop.examples.terasort.TeraValidate

    LOG.info("TeraSort took {} ms", stopWatch.getTime());
  }


  private void runTeraValidate() throws Exception {
    StopWatch stopWatch = new StopWatch();
    TeraValidate teraValidate = new TeraValidate();
    teraValidate.setConf(controller.getJobConf());
    LOG.info("Starting TeraValidate");
    stopWatch.start();
    teraValidate.run(new String[] { "output", "report" });
    stopWatch.stop();
    LOG.info("TeraValidate took {} ms", stopWatch.getTime());
  }

View Full Code Here

Examples of org.apache.hadoop.examples.terasort.TeraValidate

    LOG.info("TeraSort took {} ms", stopWatch.getTime());
  }


  private void runTeraValidate() throws Exception {
    StopWatch stopWatch = new StopWatch();
    TeraValidate teraValidate = new TeraValidate();
    teraValidate.setConf(controller.getJobConf());
    LOG.info("Starting TeraValidate");
    stopWatch.start();
    teraValidate.run(new String[] { "output", "report" });
    stopWatch.stop();
    LOG.info("TeraValidate took {} ms", stopWatch.getTime());
  }

View Full Code Here

Examples of org.apache.tez.mapreduce.examples.terasort.TeraValidate

                                       Path sortOutput, Path valOutput) 
  throws Exception {
    String[] svArgs = {sortOutput.toString(), valOutput.toString()};


    // Run Tera-Validator
    assertEquals(ToolRunner.run(job, new TeraValidate(), svArgs), 0);
  }

View Full Code Here

Examples of org.apache.tez.mapreduce.examples.terasort.TeraValidate

                                       Path sortOutput, Path valOutput) 
  throws Exception {
    String[] svArgs = {sortOutput.toString(), valOutput.toString()};


    // Run Tera-Validator
    assertEquals(ToolRunner.run(job, new TeraValidate(), svArgs), 0);
  }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.