List of commitJob() Examples

Examples of commitJob()

org.apache.hadoop.hive.shims.HadoopShims.HCatHadoopShims.commitJob()
org.apache.hadoop.mapreduce.OutputCommitter.commitJob()
For committing job's output after successful job completion. Note that this is invoked for jobs with final runstate as SUCCESSFUL. This is called from the application master process for the entire job. This is guaranteed to only be called once. If it throws an exception the entire job will fail. @param jobContext Context of the job whose output is being written. @throws IOException

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

      }
    } finally {
      writer.close(context);
    }
    committer.commitTask(context);
    committer.commitJob(job);


    InputFormat<IntWritable, DoubleWritable> iformat =
      new SequenceFileInputFormat<IntWritable, DoubleWritable>();
    int count = 0;
    r.setSeed(seed);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

                }


                @Override
                public void commitJob(JobContext jobContext) throws IOException {
                    try {
                        baseOutputCommitter.commitJob(jobContext);
                        Configuration conf = jobContext.getConfiguration();
                        try {
                        //import hfiles
                        new LoadIncrementalHFiles(conf)
                                .doBulkLoad(HFileOutputFormat.getOutputPath(jobContext),

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

    publishTest(job);
  }


  public void publishTest(Job job) throws Exception {
    OutputCommitter committer = new FileOutputCommitterContainer(job, null);
    committer.commitJob(job);


    Partition part = client.getPartition(dbName, tblName, Arrays.asList("p1"));
    assertNotNull(part);


    StorerInfo storer = InternalUtil.extractStorerInfo(part.getSd(), part.getParameters());

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

        }


        @Override
        public void commitJob(JobContext jobContext) throws IOException {
          try {
            baseOutputCommitter.commitJob(jobContext);
            Configuration conf = jobContext.getConfiguration();
            try {
              //import hfiles
              new LoadIncrementalHFiles(conf)
                .doBulkLoad(HFileOutputFormat.getOutputPath(jobContext),

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

      }
    } finally {
      if (rw != null) {
        rw.close(attemptContext);
        committer.commitTask(attemptContext);
        committer.commitJob(job);
      }
    }


    byte[] result = md5.digest();
    md5.reset();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

      }
    } finally {
      writer.close(context);
    }
    committer.commitTask(context);
    committer.commitJob(job);


    InputFormat<IntWritable, DoubleWritable> iformat =
      new SequenceFileInputFormat<IntWritable, DoubleWritable>();
    int count = 0;
    r.setSeed(seed);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

        job.getConfiguration(), ShimLoader.getHadoopShims().getHCatShim().createTaskAttemptID());
    OutputCommitter committer = hcof.getOutputCommitter(tac);
    committer.setupJob(job);
    committer.setupTask(tac);
    committer.commitTask(tac);
    committer.commitJob(job);


    Partition part = client.getPartition(dbName, tblName, Arrays.asList("p1"));
    assertNotNull(part);


    StorerInfo storer = InternalUtil.extractStorerInfo(part.getSd(), part.getParameters());

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

      }
    } finally {
      writer.close(context);
    }
    committer.commitTask(context);
    committer.commitJob(job);


    InputFormat<IntWritable, DoubleWritable> iformat =
      new SequenceFileInputFormat<IntWritable, DoubleWritable>();
    int count = 0;
    r.setSeed(seed);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

    TaskAttemptContext
        taskContext = new HackTaskAttemptContext(jobConf, new TaskAttemptID());
    OutputCommitter outputCommitter = outputFormat.getOutputCommitter(
        taskContext);
    JobContext jobContext = new HackJobContext(jobConf, taskContext.getJobID());
    outputCommitter.commitJob(jobContext);
  }


  public static JobContext makeJobContext(Configuration conf) {
    JobConf jobConf = new JobConf(conf);
    TaskAttemptContext

View Full Code Here

Examples of org.apache.hadoop.mapreduce.OutputCommitter.commitJob()

      }


      /*if_not[HADOOP_NON_COMMIT_JOB]*/
      @Override
      public void commitJob(JobContext context) throws IOException {
        outputCommitter.commitJob(
            HadoopUtils.makeJobContext(getConf(), context));
      }


      @Override
      public void abortJob(JobContext context,

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.