List of commitTask() Examples

Examples of commitTask()

org.apache.hadoop.mapreduce.OutputCommitter.commitTask()
To promote the task's temporary output to final output location. If {@link #needsTaskCommit(TaskAttemptContext)} returns true and thistask is the task that the AM determines finished first, this method is called to commit an individual task's output. This is to mark that tasks output as complete, as {@link #commitJob(JobContext)} will also be called later on if the entire job finished successfully. This is called from a task's process. This may be called multiple times for the same task, but different task attempts. It should be very rare for this to be called multiple times and requires odd networking failures to make this happen. In the future the Hadoop framework may eliminate this race. @param taskContext Context of the task whose output is being written. @throws IOException if commit is not successful.
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask()
Move the files from the work directory to the job output directory @param context the task context

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()


      while(rr.nextKeyValue()){
        rw.write(rr.getCurrentKey(), rr.getCurrentValue());
      }
      rw.close(wtaskContext);
      taskOc.commitTask(wtaskContext);
      rr.close();
    }


    oc.commitJob(ojob);
  }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        md5.update(value.getBytes(), 0, value.getLength());
      }
    } finally {
      if (rw != null) {
        rw.close(attemptContext);
        committer.commitTask(attemptContext);
        committer.commitJob(job);
      }
    }


    byte[] result = md5.digest();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

  }
  
  OutputFormat outputFormat = ReflectionUtils.newInstance(
      tContext.getOutputFormatClass(), conf);
  OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
  committer.commitTask(tContext);
}
  
  
  private void writeOutput(TaskAttempt attempt, Configuration conf)
    throws Exception {

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    }
    
    OutputFormat outputFormat = ReflectionUtils.newInstance(
        tContext.getOutputFormatClass(), conf);
    OutputCommitter committer = outputFormat.getOutputCommitter(tContext);
    committer.commitTask(tContext);
  }


  private void validateOutput() throws IOException {
    File expectedFile = new File(new Path(outputDir, partFile).toString());
    StringBuffer expectedOutput = new StringBuffer();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        writer.write(bkey, bval);
      }
    } finally {
      writer.close(context);
    }
    committer.commitTask(context);
    committer.commitJob(job);


    InputFormat<IntWritable, DoubleWritable> iformat =
      new SequenceFileInputFormat<IntWritable, DoubleWritable>();
    int count = 0;

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        HCatRecord rec = recordItr.next();
        writer.write(null, rec);
      }
      writer.close(cntxt);
      if (committer.needsTaskCommit(cntxt)) {
        committer.commitTask(cntxt);
      }
    } catch (IOException e) {
      if (null != committer) {
        try {
          committer.abortTask(cntxt);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

        BaseOutputCommitterContainer outputContainer = outputCommitters.get(alias);
        OutputCommitter baseCommitter = outputContainer.getBaseCommitter();
        TaskAttemptContext committerContext = outputContainer.getContext();
        if (baseCommitter.needsTaskCommit(committerContext)) {
          LOGGER.info("Calling commitTask for alias: " + alias);
          baseCommitter.commitTask(committerContext);
        }
      }
    }


    @Override

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

          OutputJobInfo outputJobInfo = outputJobInfoEntry.getValue();
          LOG.info("Committing task-attempt for " + outputJobInfo.getLocation());
          TaskAttemptContext dynContext = dynamicContexts.get(dynKey);
          OutputCommitter dynCommitter = baseDynamicCommitters.get(dynKey);
          if (dynCommitter.needsTaskCommit(dynContext)) {
            dynCommitter.commitTask(dynContext);
          }
          else {
            LOG.info("Skipping commitTask() for " + outputJobInfo.getLocation());
          }
        }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()

    TaskAttemptContext tac = ShimLoader.getHadoopShims().getHCatShim().createTaskAttemptContext(
        job.getConfiguration(), ShimLoader.getHadoopShims().getHCatShim().createTaskAttemptID());
    OutputCommitter committer = hcof.getOutputCommitter(tac);
    committer.setupJob(job);
    committer.setupTask(tac);
    committer.commitTask(tac);
    committer.commitJob(job);


    Partition part = client.getPartition(dbName, tblName, Arrays.asList("p1"));
    assertNotNull(part);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitTask()


      while(rr.nextKeyValue()){
        rw.write(rr.getCurrentKey(), rr.getCurrentValue());
      }
      rw.close(wtaskContext);
      taskOc.commitTask(wtaskContext);
      rr.close();
    }


    oc.commitJob(ojob);
  }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.