List of commitTask() Examples

Examples of commitTask()

org.apache.hadoop.mapreduce.OutputCommitter.commitTask()
To promote the task's temporary output to final output location. If {@link #needsTaskCommit(TaskAttemptContext)} returns true and thistask is the task that the AM determines finished first, this method is called to commit an individual task's output. This is to mark that tasks output as complete, as {@link #commitJob(JobContext)} will also be called later on if the entire job finished successfully. This is called from a task's process. This may be called multiple times for the same task, but different task attempts. It should be very rare for this to be called multiple times and requires odd networking failures to make this happen. In the future the Hadoop framework may eliminate this race. @param taskContext Context of the task whose output is being written. @throws IOException if commit is not successful.
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask()
Move the files from the work directory to the job output directory @param context the task context

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

                    return baseOutputCommitter.needsTaskCommit(taskContext);
                }


                @Override
                public void commitTask(TaskAttemptContext taskContext) throws IOException {
                    baseOutputCommitter.commitTask(taskContext);
                }


                @Override
                public void abortTask(TaskAttemptContext taskContext) throws IOException {
                    baseOutputCommitter.abortTask(taskContext);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

                BaseOutputCommitterContainer outputContainer = outputCommitters.get(alias);
                OutputCommitter baseCommitter = outputContainer.getBaseCommitter();
                TaskAttemptContext committerContext = outputContainer.getContext();
                if (baseCommitter.needsTaskCommit(committerContext)) {
                    LOGGER.info("Calling commitTask for alias: " + alias);
                    baseCommitter.commitTask(committerContext);
                }
            }
        }


        @Override

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

            }
            for (Map.Entry<String, org.apache.hadoop.mapred.OutputCommitter> entry : baseDynamicCommitters.entrySet()) {
                org.apache.hadoop.mapred.TaskAttemptContext currContext = dynamicContexts.get(entry.getKey());
                OutputCommitter baseOutputCommitter = entry.getValue();
                if (baseOutputCommitter.needsTaskCommit(currContext)) {
                    baseOutputCommitter.commitTask(currContext);
                }
            }
        } else {
            getBaseRecordWriter().close(reporter);
        }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

                HCatRecord rec = recordItr.next();
                writer.write(null, rec);
            }
            writer.close(cntxt);
            if (committer.needsTaskCommit(cntxt)) {
                committer.commitTask(cntxt);
            }
        } catch (IOException e) {
            if (null != committer) {
                try {
                    committer.abortTask(cntxt);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        if (outputCommitter.needsTaskCommit(getContext())) {
          LoggerUtils.setStatusAndLog(getContext(), LOG, Level.INFO,
            "OutputCommitter: committing task output.");
          // transfer from temp dirs to "task commit" dirs to prep for
          // the master's OutputCommitter#commitJob(context) call to finish.
          outputCommitter.commitTask(getContext());
        }
      } catch (InterruptedException ie) {
        LOG.error("Interrupted while attempting to obtain " +
          "OutputCommitter.", ie);
      } catch (IOException ioe) {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

  }
  
  OutputFormat outputFormat = ReflectionUtils.newInstance(
      tContext.getOutputFormatClass(), conf);
  OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
  committer.commitTask(tContext);
}
  
  
  private void writeOutput(TaskAttempt attempt, Configuration conf)
    throws Exception {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    }
    
    OutputFormat outputFormat = ReflectionUtils.newInstance(
        tContext.getOutputFormatClass(), conf);
    OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
    committer.commitTask(tContext);
  }


  private void validateOutput() throws IOException {
    File expectedFile = new File(new Path(outputDir, partFile).toString());
    StringBuffer expectedOutput = new StringBuffer();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

            rw.close(tac);
        } catch (InterruptedException e) {
            throw new IOException(e);
        }
        if(oc.needsTaskCommit(tac)) {
            oc.commitTask(tac);
        }
        HadoopShims.commitOrCleanup(oc, jc);
    }


    @Override

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

  }
  
  OutputFormat outputFormat = ReflectionUtils.newInstance(
      tContext.getOutputFormatClass(), conf);
  OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
  committer.commitTask(tContext);
}
  
  
  private void writeOutput(TaskAttempt attempt, Configuration conf)
    throws Exception {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    }
    
    OutputFormat outputFormat = ReflectionUtils.newInstance(
        tContext.getOutputFormatClass(), conf);
    OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
    committer.commitTask(tContext);
  }


  private void validateOutput() throws IOException {
    File expectedFile = new File(new Path(outputDir, partFile).toString());
    StringBuffer expectedOutput = new StringBuffer();

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.