Examples of TeraGen

org.apache.hadoop.examples.terasort.TeraGen
Generate the official GraySort input data set. The user specifies the number of rows and the output directory and this class runs a map/reduce program to generate the data. The format of the data is:
- (10 bytes key) (constant 2 bytes) (32 bytes rowid) (constant 4 bytes) (48 bytes filler) (constant 4 bytes)
- The rowid is the right justified row id as a hex number.
To run the program: bin/hadoop jar hadoop-*-examples.jar teragen 10000000000 in-dir
org.apache.tez.mapreduce.examples.terasort.TeraGen
Generate the official GraySort input data set. The user specifies the number of rows and the output directory and this class runs a map/reduce program to generate the data. The format of the data is:
- (10 bytes key) (constant 2 bytes) (32 bytes rowid) (constant 4 bytes) (48 bytes filler) (constant 4 bytes)
- The rowid is the right justified row id as a hex number.
To run the program: bin/hadoop jar hadoop-*-examples.jar teragen 10000000000 in-dir

Examples of org.apache.hadoop.examples.terasort.TeraGen

    int numTaskTrackers = controller.getCluster().getInstances().size() - 1;
    long bytesPerNode =
      Long.parseLong(System.getProperty("terasortBytesPerNode", "1000000000"));
    long rows = numTaskTrackers * (bytesPerNode / 100);
    StopWatch stopWatch = new StopWatch();
    TeraGen teraGen = new TeraGen();
    teraGen.setConf(controller.getJobConf());
    LOG.info("Starting TeraGen with {} tasktrackers, {} bytes per node, {} rows",
        new Object[] { numTaskTrackers, bytesPerNode, rows});
    stopWatch.start();
    teraGen.run(new String[] { "" + rows, "input" });
    stopWatch.stop();
    LOG.info("TeraGen took {} ms", stopWatch.getTime());
  }

View Full Code Here

Examples of org.apache.hadoop.examples.terasort.TeraGen

    int numTaskTrackers = controller.getCluster().getInstances().size() - 1;
    long bytesPerNode =
      Long.parseLong(System.getProperty("terasortBytesPerNode", "1000000000"));
    long rows = numTaskTrackers * (bytesPerNode / 100);
    StopWatch stopWatch = new StopWatch();
    TeraGen teraGen = new TeraGen();
    teraGen.setConf(controller.getJobConf());
    LOG.info("Starting TeraGen with {} tasktrackers, {} bytes per node, {} rows",
        new Object[] { numTaskTrackers, bytesPerNode, rows});
    stopWatch.start();
    teraGen.run(new String[] { "" + rows, "input" });
    stopWatch.stop();
    LOG.info("TeraGen took {} ms", stopWatch.getTime());
  }

View Full Code Here

Examples of org.apache.tez.mapreduce.examples.terasort.TeraGen

  private void runTeraGen(Configuration conf, Path sortInput) 
      throws Exception {
    String[] genArgs = {NUM_ROWS, sortInput.toString()};
    
    // Run TeraGen
    assertEquals(ToolRunner.run(conf, new TeraGen(), genArgs), 0);
  }

View Full Code Here

Examples of org.apache.tez.mapreduce.examples.terasort.TeraGen

  private void runTeraGen(Configuration conf, Path sortInput) 
      throws Exception {
    String[] genArgs = {NUM_ROWS, sortInput.toString()};
    
    // Run TeraGen
    assertEquals(ToolRunner.run(conf, new TeraGen(), genArgs), 0);
  }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.