Examples of JobClient

com.asakusafw.yaess.jobqueue.client.JobClient
An abstract interface of job queue client. @since 0.2.6
eu.stratosphere.nephele.client.JobClient
The job client is able to submit, control, and abort jobs.
This class is thread-safe.
org.apache.flink.runtime.client.JobClient
The job client is able to submit, control, and abort jobs.
org.apache.hadoop.mapred.JobClient
te a new JobConf JobConf job = new JobConf(new Configuration(), MyJob.class); // Specify various job-specific parameters job.setJobName("myjob"); job.setInputPath(new Path("in")); job.setOutputPath(new Path("out")); job.setMapperClass(MyJob.MyMapper.class); job.setReducerClass(MyJob.MyReducer.class); // Submit the job, then poll for progress until the job is complete JobClient.runJob(job);

Job Control

At times clients would chain map-reduce jobs to accomplish complex tasks which cannot be done via a single map-reduce job. This is fairly easy since the output of the job, typically, goes to distributed file-system and that can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such situations the various job-control options are:
1. {@link #runJob(JobConf)} : submits the job and returns only after the job has completed.
2. {@link #submitJob(JobConf)} : only submits the job, then poll the returned handle to the {@link RunningJob} to query status and make scheduling decisions.
3. {@link JobConf#setJobEndNotificationURI(String)} : setup a notificationon job-completion, thus avoiding polling.
@see JobConf @see ClusterStatus @see Tool @see DistributedCache
org.apache.nutch.mapReduce.JobClient
JobClient interacts with the JobTracker network interface. This object implements the job-control interface, and should be the primary method by which user programs interact with the networked job system. @author Mike Cafarella

Examples of org.apache.hadoop.mapred.JobClient

  @Override
  public synchronized void connect() throws IOException {
    if (isConnected()) {
      return;
    }
    client = new JobClient(new JobConf(getConf()));
    setConnected(true);
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    }
    conf.set(DST_DIR_LABEL, outputPath.toString());
    final String randomId = DistCp.getRandomId();
    Path stagingArea;
    try {
      stagingArea = JobSubmissionFiles.getStagingDir(new JobClient(conf),
            conf);
    } catch (InterruptedException e) {
      throw new IOException(e);
    }
    Path jobDirectory = new Path(stagingArea,

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    for (String propertyKey : properties.keySet())
      conf.set(propertyKey, properties.get(propertyKey));


    configJob(conf);


    JobClient c = new JobClient(conf);
    RunningJob job = c.runJob(conf);
    return job.isSuccessful();
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

      File wd = new File(".").getAbsoluteFile();
      StreamUtil.unJar(new File(jar_), wd);
    }


    // if jobConf_ changes must recreate a JobClient
    jc_ = new JobClient(jobConf_);
    boolean error = true;
    running_ = null;
    String lastReport = null;
    try {
      running_ = jc_.submitJob(jobConf_);

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

      throws IOException {
    int numMaps =
      (int)(totalBytes / job.getLong(BYTES_PER_MAP_LABEL, BYTES_PER_MAP));
    numMaps = Math.min(numMaps, 
        job.getInt(MAX_MAPS_LABEL, MAX_MAPS_PER_NODE *
          new JobClient(job).getClusterStatus().getTaskTrackers()));
    job.setNumMapTasks(Math.max(numMaps, 1));
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

        args.flags.contains(Options.IGNORE_READ_FAILURES));
    jobConf.setBoolean(Options.PRESERVE_STATUS.propertyname,
        args.flags.contains(Options.PRESERVE_STATUS));


    final String randomId = getRandomId();
    JobClient jClient = new JobClient(jobConf);
    Path stagingArea;
    try {
      stagingArea = JobSubmissionFiles.getStagingDir(jClient, conf);
    } catch (InterruptedException e) {
      throw new IOException(e);
    }
    
    Path jobDirectory = new Path(stagingArea + NAME + "_" + randomId);
    FsPermission mapredSysPerms =
      new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION);
    FileSystem.mkdirs(jClient.getFs(), jobDirectory, mapredSysPerms);
    jobConf.set(JOB_DIR_LABEL, jobDirectory.toString());


    long maxBytesPerMap = conf.getLong(BYTES_PER_MAP_LABEL, BYTES_PER_MAP);


    FileSystem dstfs = args.dst.getFileSystem(conf);

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    this.state = Job.WAITING;
    this.jobID = "unassigned";
    this.mapredJobID = null; //not yet assigned 
    this.jobName = jobConf.getJobName();
    this.message = "just initialized";
    this.jc = new JobClient(jobConf);
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    if (args.length == 0) {
      return printUsage();    
    }
    
    Configuration conf = getConf();
    JobClient client = new JobClient(conf);
    ClusterStatus cluster = client.getClusterStatus();
    int numMapsPerHost = conf.getInt(MAPS_PER_HOST, 10);
    long numBytesToWritePerMap = conf.getLong(BYTES_PER_MAP,
                                             1*1024*1024*1024);
    if (numBytesToWritePerMap == 0) {
      System.err.println("Cannot have " + BYTES_PER_MAP +" set to 0");

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

                                  String modelPrototype,
                                  int prototypeSize,
                                  int numClusters,
                                  double alpha_0,
                                  int numReducers) {
    Configurable client = new JobClient();
    JobConf conf = new JobConf(DirichletDriver.class);
    
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(DirichletCluster.class);
    conf.setMapOutputKeyClass(Text.class);
    conf.setMapOutputValueClass(VectorWritable.class);
    
    FileInputFormat.setInputPaths(conf, new Path(input));
    Path outPath = new Path(stateOut);
    FileOutputFormat.setOutputPath(conf, outPath);
    
    conf.setMapperClass(DirichletMapper.class);
    conf.setReducerClass(DirichletReducer.class);
    conf.setNumReduceTasks(numReducers);
    conf.setInputFormat(SequenceFileInputFormat.class);
    conf.setOutputFormat(SequenceFileOutputFormat.class);
    conf.set(STATE_IN_KEY, stateIn);
    conf.set(MODEL_FACTORY_KEY, modelFactory);
    conf.set(MODEL_PROTOTYPE_KEY, modelPrototype);
    conf.set(PROTOTYPE_SIZE_KEY, Integer.toString(prototypeSize));
    conf.set(NUM_CLUSTERS_KEY, Integer.toString(numClusters));
    conf.set(ALPHA_0_KEY, Double.toString(alpha_0));
    
    client.setConf(conf);
    try {
      JobClient.runJob(conf);
    } catch (IOException e) {
      log.warn(e.toString(), e);
    }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

   *          the directory pathname for input state
   * @param output
   *          the directory pathname for output points
   */
  public static void runClustering(String input, String stateIn, String output) {
    Configurable client = new JobClient();
    JobConf conf = new JobConf(DirichletDriver.class);
    
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(Text.class);
    
    FileInputFormat.setInputPaths(conf, new Path(input));
    Path outPath = new Path(output);
    FileOutputFormat.setOutputPath(conf, outPath);
    
    conf.setMapperClass(DirichletMapper.class);
    conf.setNumReduceTasks(0);
    
    client.setConf(conf);
    try {
      JobClient.runJob(conf);
    } catch (IOException e) {
      log.warn(e.toString(), e);
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of JobClient

Job Control

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient