List of commitTask() Examples

Examples of commitTask()

org.apache.hadoop.mapreduce.OutputCommitter.commitTask()
To promote the task's temporary output to final output location. If {@link #needsTaskCommit(TaskAttemptContext)} returns true and thistask is the task that the AM determines finished first, this method is called to commit an individual task's output. This is to mark that tasks output as complete, as {@link #commitJob(JobContext)} will also be called later on if the entire job finished successfully. This is called from a task's process. This may be called multiple times for the same task, but different task attempts. It should be very rare for this to be called multiple times and requires odd networking failures to make this happen. In the future the Hadoop framework may eliminate this race. @param taskContext Context of the task whose output is being written. @throws IOException if commit is not successful.
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask()
Move the files from the work directory to the job output directory @param context the task context

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    public void close() throws IOException {
        LOG.debug("Committing output results: {}", format.getClass().getName());
        try {
            writer.close(context);
            OutputCommitter comitter = format.getOutputCommitter(context);
            comitter.commitTask(context);
            comitter.commitJob(context);
        } catch (InterruptedException e) {
            throw (InterruptedIOException) new InterruptedIOException().initCause(e);
        }
    }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

            rw.close(tac);
        } catch (InterruptedException e) {
            throw new IOException(e);
        }
        if(oc.needsTaskCommit(tac)) {
            oc.commitTask(tac);
        }
        oc.cleanupJob(jc);
    }


    @Override

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        HCatRecord rec = recordItr.next();
        writer.write(null, rec);
      }
      writer.close(cntxt);
      if (committer.needsTaskCommit(cntxt)) {
        committer.commitTask(cntxt);
      }
    } catch (IOException e) {
      if (null != committer) {
        try {
          committer.abortTask(cntxt);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()


      while(rr.nextKeyValue()){
        rw.write(rr.getCurrentKey(), rr.getCurrentValue());
      }
      rw.close(wtaskContext);
      taskOc.commitTask(wtaskContext);
      rr.close();
    }


    oc.commitJob(ojob);
  }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

          return baseOutputCommitter.needsTaskCommit(taskContext);
        }


        @Override
        public void commitTask(TaskAttemptContext taskContext) throws IOException {
          baseOutputCommitter.commitTask(taskContext);
        }


        @Override
        public void abortTask(TaskAttemptContext taskContext) throws IOException {
          baseOutputCommitter.abortTask(taskContext);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

          bwriter.close(context);
        }
        for (HCatOutputStorageDriver osd : baseDynamicStorageDrivers.values()){
          OutputCommitter baseOutputCommitter = osd.getOutputFormat().getOutputCommitter(context);
          if (baseOutputCommitter.needsTaskCommit(context)){
            baseOutputCommitter.commitTask(context);
          }
        }
      } else {
        baseWriter.close(context);
      }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

  }
  
  OutputFormat outputFormat = ReflectionUtils.newInstance(
      tContext.getOutputFormatClass(), conf);
  OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
  committer.commitTask(tContext);
}
  
  
  private void writeOutput(TaskAttempt attempt, Configuration conf)
    throws Exception {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    }
    
    OutputFormat outputFormat = ReflectionUtils.newInstance(
        tContext.getOutputFormatClass(), conf);
    OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
    committer.commitTask(tContext);
  }


  private void validateOutput() throws IOException {
    File expectedFile = new File(new Path(outputDir, partFile).toString());
    StringBuffer expectedOutput = new StringBuffer();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

  }
  
  OutputFormat outputFormat = ReflectionUtils.newInstance(
      tContext.getOutputFormatClass(), conf);
  OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
  committer.commitTask(tContext);
}
  
  
  private void writeOutput(TaskAttempt attempt, Configuration conf)
    throws Exception {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    }
    
    OutputFormat outputFormat = ReflectionUtils.newInstance(
        tContext.getOutputFormatClass(), conf);
    OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
    committer.commitTask(tContext);
  }


  private void validateOutput() throws IOException {
    File expectedFile = new File(new Path(outputDir, partFile).toString());
    StringBuffer expectedOutput = new StringBuffer();

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.