Examples of org.apache.hadoop.mapred.OutputCommitter

org.apache.hadoop.mapred.OutputCommitter
OutputCommitter describes the commit of task output for a Map-Reduce job.
The Map-Reduce framework relies on the OutputCommitter of the job to:
1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion.
3. Setup the task temporary output.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output.
6. Discard the task commit.
@see FileOutputCommitter @see JobContext @see TaskAttemptContext @deprecated Use {@link org.apache.hadoop.mapreduce.OutputCommitter} instead.

    throws IOException, InterruptedException
  {
    HadoopUtils.setWorkOutputDir(taskAttemptContext);
    Configuration conf = taskAttemptContext.getConfiguration();
    JobConf jobConf = new JobConf(conf);
    OutputCommitter baseCommitter = jobConf.getOutputCommitter();
    LOG.info("Getting output committer with base output committer {}",
        baseCommitter.getClass().getSimpleName());
    return new HiveApiOutputCommitter(new HackOutputCommitter(baseCommitter, jobConf), myProfileId);
  }

View Full Code Here

    throws IOException, InterruptedException
  {
    HadoopUtils.setWorkOutputDir(taskAttemptContext);
    Configuration conf = taskAttemptContext.getConfiguration();
    JobConf jobConf = new JobConf(conf);
    OutputCommitter baseCommitter = jobConf.getOutputCommitter();
    LOG.info("Getting output committer with base output committer {}",
        baseCommitter.getClass().getSimpleName());
    return new HiveApiOutputCommitter(new HackOutputCommitter(baseCommitter, jobConf), myProfileId);
  }

View Full Code Here

    throws IOException, InterruptedException
  {
    HadoopUtils.setWorkOutputDir(taskAttemptContext);
    Configuration conf = taskAttemptContext.getConfiguration();
    JobConf jobConf = new JobConf(conf);
    OutputCommitter baseCommitter = jobConf.getOutputCommitter();
    LOG.info("Getting output committer with base output committer {}",
        baseCommitter.getClass().getSimpleName());
    return new HiveApiOutputCommitter(new HackOutputCommitter(baseCommitter, jobConf), myProfileId);
  }

View Full Code Here

    throws IOException, InterruptedException
  {
    HadoopUtils.setWorkOutputDir(taskAttemptContext);
    Configuration conf = taskAttemptContext.getConfiguration();
    JobConf jobConf = new JobConf(conf);
    OutputCommitter baseCommitter = jobConf.getOutputCommitter();
    LOG.info("Getting output committer with base output committer {}",
        baseCommitter.getClass().getSimpleName());
    return new HiveApiOutputCommitter(new HackOutputCommitter(baseCommitter, jobConf), myProfileId);
  }

View Full Code Here

TOP

Related Classes of org.apache.hadoop.mapred.OutputCommitter

com.facebook.hiveio.output.HiveApiOutputFormat

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.