Examples of OutputCommitter

org.apache.hadoop.mapred.OutputCommitter
OutputCommitter describes the commit of task output for a Map-Reduce job.
The Map-Reduce framework relies on the OutputCommitter of the job to:
1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion.
3. Setup the task temporary output.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output.
6. Discard the task commit.
@see FileOutputCommitter @see JobContext @see TaskAttemptContext @deprecated Use {@link org.apache.hadoop.mapreduce.OutputCommitter} instead.
org.apache.hadoop.mapreduce.OutputCommitter
OutputCommitter describes the commit of task output for a Map-Reduce job.
The Map-Reduce framework relies on the OutputCommitter of the job to:
1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion.
3. Setup the task temporary output.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output.
6. Discard the task commit.
The methods in this class can be called from several different processes and from several different contexts. It is important to know which process and which context each is called from. Each method should be marked accordingly in its documentation. It is also important to note that not all methods are guaranteed to be called once and only once. If a method is not guaranteed to have this property the output committer needs to handle this appropriately. Also note it will only be in rare situations where they may be called multiple times for the same task. @see org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter @see JobContext @see TaskAttemptContext
org.apache.tez.runtime.api.OutputCommitter
OutputCommitter to "finalize" the output and make it user-visible if needed. The OutputCommitter is allowed only on a terminal Output.

Examples of org.apache.hadoop.mapred.OutputCommitter

    throws IOException, InterruptedException
  {
    HadoopUtils.setWorkOutputDir(taskAttemptContext);
    Configuration conf = taskAttemptContext.getConfiguration();
    JobConf jobConf = new JobConf(conf);
    OutputCommitter baseCommitter = jobConf.getOutputCommitter();
    LOG.info("Getting output committer with base output committer {}",
        baseCommitter.getClass().getSimpleName());
    return new HiveApiOutputCommitter(new HackOutputCommitter(baseCommitter, jobConf), myProfileId);
  }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

    private OutputCommitter createOutputCommitter(
            TaskAttemptContext context) throws IOException, InterruptedException {
        assert context != null;
        Set<OutputCommitter> components = new LinkedHashSet<OutputCommitter>();
        if (isBridgeOutputEnabled(context)) {
            OutputCommitter committer = bridgeOutputFormat.getOutputCommitter(context);
            if (components.contains(committer) == false) {
                components.add(committer);
            }
        }
        if (isFileOutputEnabled(context)) {
            OutputCommitter committer = dummyFileOutputFormat.getOutputCommitter(context);
            if (components.contains(committer) == false) {
                components.add(committer);
            }
        }
        if (isTemporaryOutputEnabled(context)) {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

    @Override
    public void close() throws IOException {
        LOG.debug("Committing output results: {}", format.getClass().getName());
        try {
            writer.close(context);
            OutputCommitter comitter = format.getOutputCommitter(context);
            comitter.commitTask(context);
            comitter.commitJob(context);
        } catch (InterruptedException e) {
            throw (InterruptedIOException) new InterruptedIOException().initCause(e);
        }
    }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

    when(appContext.getApplicationID()).thenReturn(attemptid.getApplicationId());
    when(appContext.getApplicationAttemptId()).thenReturn(attemptid);
    when(appContext.getEventHandler()).thenReturn(
        dispatcher.getEventHandler());
    when(appContext.getClock()).thenReturn(clock);
    OutputCommitter committer = mock(OutputCommitter.class);
    TestingRMHeartbeatHandler rmhh =
        new TestingRMHeartbeatHandler();


    CommitterEventHandler ceh = new CommitterEventHandler(appContext,
        committer, rmhh);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

  }


  @Test
  public void testBasic() throws Exception {
    AppContext mockContext = mock(AppContext.class);
    OutputCommitter mockCommitter = mock(OutputCommitter.class);
    Clock mockClock = mock(Clock.class);
    
    CommitterEventHandler handler = new CommitterEventHandler(mockContext, 
        mockCommitter, new TestingRMHeartbeatHandler());
    YarnConfiguration conf = new YarnConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

  }


  @Test
  public void testFailure() throws Exception {
    AppContext mockContext = mock(AppContext.class);
    OutputCommitter mockCommitter = mock(OutputCommitter.class);
    Clock mockClock = mock(Clock.class);
    
    CommitterEventHandler handler = new CommitterEventHandler(mockContext, 
        mockCommitter, new TestingRMHeartbeatHandler());
    YarnConfiguration conf = new YarnConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

    when(appContext.getApplicationID()).thenReturn(attemptid.getApplicationId());
    when(appContext.getApplicationAttemptId()).thenReturn(attemptid);
    when(appContext.getEventHandler()).thenReturn(
        dispatcher.getEventHandler());
    when(appContext.getClock()).thenReturn(clock);
    OutputCommitter committer = mock(OutputCommitter.class);
    TestingRMHeartbeatHandler rmhh =
        new TestingRMHeartbeatHandler();


    CommitterEventHandler ceh = new CommitterEventHandler(appContext,
        committer, rmhh);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

  }


  @Test
  public void testBasic() throws Exception {
    AppContext mockContext = mock(AppContext.class);
    OutputCommitter mockCommitter = mock(OutputCommitter.class);
    Clock mockClock = mock(Clock.class);
    
    CommitterEventHandler handler = new CommitterEventHandler(mockContext, 
        mockCommitter, new TestingRMHeartbeatHandler());
    YarnConfiguration conf = new YarnConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

  }


  @Test
  public void testFailure() throws Exception {
    AppContext mockContext = mock(AppContext.class);
    OutputCommitter mockCommitter = mock(OutputCommitter.class);
    Clock mockClock = mock(Clock.class);
    
    CommitterEventHandler handler = new CommitterEventHandler(mockContext, 
        mockCommitter, new TestingRMHeartbeatHandler());
    YarnConfiguration conf = new YarnConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter

    
 
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    OutputCommitter committer = mock(OutputCommitter.class);
    CommitterEventHandler commitHandler =
        createCommitterEventHandler(dispatcher, committer);
    commitHandler.init(conf);
    commitHandler.start();

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.