Examples of TaskSpec

edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec
org.apache.tez.runtime.api.impl.TaskSpec
org.rlcommunity.rlglue.codec.taskspec.TaskSpec
cs.ualberta.ca/RLBB/TaskSpecification.html"> RL-Glue Task Spec info
This class was written to handle future revisions to the Task Spec while providing backwards compatibility with old Task Spec versions. This is accomplished through the use of the factory design pattern. The TaskSpecDelegate object is the medium through which the TaskSpec communicates with different implementations of the Task Spec versions. So far, each Task Spec version has incrementally added functionality, but this might not always be the case, so this framework was designed to robustly accomodate future versions.
This most recent Implementation of TaskSpecDelegate is TaskSpecVRLGLUE3
@author Matt Radkie

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec


    // Make sure that tasks are launched right away, if resources are available.
    scheduler.submitTaskReservations(createTaskReservationRequest(1, scheduler, testApp),
                                     backendAddress);
    assertEquals(1, scheduler.runnableTasks());
    TaskSpec task = scheduler.getNextTask();
    assertEquals("1", task.requestId);
    assertEquals(0, scheduler.runnableTasks());


    scheduler.submitTaskReservations(createTaskReservationRequest(2, scheduler, testApp),
                                     backendAddress);

View Full Code Here

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

    completedTasks.add(fullTaskId);
    fullTaskId.requestId = "1";


    scheduler.tasksFinished(completedTasks);
    assertEquals(1, scheduler.runnableTasks());
    TaskSpec task = scheduler.getNextTask();
    assertEquals("5", task.requestId);
    assertEquals(0, scheduler.runnableTasks());


    fullTaskId.requestId = task.requestId;
    scheduler.tasksFinished(completedTasks);

View Full Code Here

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

    scheduler.getNextTask();
    assertEquals(0, scheduler.runnableTasks());


    scheduler.submitTaskReservations(createTaskReservationRequest(1, scheduler, user2), address2);
    assertEquals(1, scheduler.runnableTasks());
    TaskSpec user2Task = scheduler.getNextTask();
    assertEquals(0, scheduler.runnableTasks());


    scheduler.submitTaskReservations(createTaskReservationRequest(1, scheduler, user3), address3);
    assertEquals(1, scheduler.runnableTasks());
    scheduler.getNextTask();
    assertEquals(0, scheduler.runnableTasks());


    scheduler.submitTaskReservations(createTaskReservationRequest(1, scheduler, user4), address4);
    assertEquals(1, scheduler.runnableTasks());
    scheduler.getNextTask();
    assertEquals(0, scheduler.runnableTasks());


    /* Create the following backlogs.
     * user1: 2 tasks
     * user2: 3 tasks
     * user3: 4 tasks
     */
    scheduler.submitTaskReservations(
        createTaskReservationRequest(2, scheduler, user1), address1);
    scheduler.submitTaskReservations(
        createTaskReservationRequest(1, scheduler, user2), address2);
    scheduler.submitTaskReservations(
        createTaskReservationRequest(1, scheduler, user2), address2);
    scheduler.submitTaskReservations(
        createTaskReservationRequest(1, scheduler, user2), address2);
    scheduler.submitTaskReservations(
        createTaskReservationRequest(4, scheduler, user3), address3);


    assertEquals(0, scheduler.runnableTasks());


    // If the getTask() for user 2 fails, the scheduler should try to get another task
    // for that user rather than using the usual round-robin ordering.
    scheduler.noTaskForReservation(user2Task);
    assertEquals(1, scheduler.runnableTasks());
    user2Task = scheduler.getNextTask();
    assertEquals(user2, user2Task.user.user);
    assertEquals("6", user2Task.requestId);
    assertEquals(0, scheduler.runnableTasks());


    // The scheduler should resume round-robin from user 1 (and not start the round-robin
    // again after user 2).
    TFullTaskId fullTaskId = new TFullTaskId();
    fullTaskId.taskId = "";
    List<TFullTaskId> completedTasks = Lists.newArrayList();
    completedTasks.add(fullTaskId);
    fullTaskId.requestId = "1";


    scheduler.tasksFinished(completedTasks);
    assertEquals(1, scheduler.runnableTasks());
    TaskSpec task = scheduler.getNextTask();
    assertEquals(user1, task.user.user);
    assertEquals("5", task.requestId);
    assertEquals(0, scheduler.runnableTasks());


    // If the scheduler eventually exhausts the queue of tasks for user 2, and cannot launch any,

View Full Code Here

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

    scheduler.submitTaskReservations(
        createTaskReservationRequest(2, scheduler, user, 1), appBackendAddress);
    scheduler.submitTaskReservations(
        createTaskReservationRequest(1, scheduler, user, 0), appBackendAddress);


    TaskSpec task = scheduler.getNextTask();
    assertEquals("1", task.requestId);
    assertEquals(3, scheduler.runnableTasks());
    task = scheduler.getNextTask();
    assertEquals("1", task.requestId);
    assertEquals(2, scheduler.runnableTasks());

View Full Code Here

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

    private HashMap<InetSocketAddress, BackendService.Client> backendClients = Maps.newHashMap();


    @Override
    public void run() {
      while (true) {
        TaskSpec task = scheduler.getNextTask(); // blocks until task is ready


        List<TTaskLaunchSpec> taskLaunchSpecs = executeGetTaskRpc(task);
        AUDIT_LOG.info(Logging.auditEventString("node_monitor_get_task_complete", task.requestId,
            nodeMonitorInternalAddress.getHost()));

View Full Code Here

Examples of org.apache.tez.runtime.api.impl.TaskSpec

        MapProcessor.class.getName()).setUserPayload(
        TezUtils.createUserPayloadFromConf(jobConf));
    
    Token<JobTokenIdentifier> shuffleToken = new Token<JobTokenIdentifier>();


    TaskSpec taskSpec = new TaskSpec(
        TezTestUtils.getMockTaskAttemptId(0, 0, mapId, 0),
        dagName, vertexName, -1,
        mapProcessorDesc,
        inputSpecs,
        outputSpecs, null);

View Full Code Here

Examples of org.apache.tez.runtime.api.impl.TaskSpec

    OutputSpec reduceOutputSpec = new OutputSpec("NullDestinationVertex",
        OutputDescriptor.create(MROutputLegacy.class.getName())
            .setUserPayload(TezUtils.createUserPayloadFromConf(jobConf)), 1);


    // Now run a reduce
    TaskSpec taskSpec = new TaskSpec(
        TezTestUtils.getMockTaskAttemptId(0, 1, 0, 0),
        dagName,
        reduceVertexName, -1,
        reduceProcessorDesc,
        Collections.singletonList(reduceInputSpec),

View Full Code Here

Examples of org.apache.tez.runtime.api.impl.TaskSpec

    TezVertexID vertexId = TezVertexID.getInstance(dagId, 1);
    TezTaskID taskId = TezTaskID.getInstance(vertexId, 1);
    TezTaskAttemptID taskAttemptId = TezTaskAttemptID.getInstance(taskId, 1);
    ProcessorDescriptor processorDescriptor = ProcessorDescriptor.create(processorClass)
        .setUserPayload(UserPayload.create(ByteBuffer.wrap(processorConf)));
    TaskSpec taskSpec = new TaskSpec(taskAttemptId, "dagName", "vertexName", -1, processorDescriptor,
        new ArrayList<InputSpec>(), new ArrayList<OutputSpec>(), null);


    TezTaskRunner taskRunner = new TezTaskRunner(tezConf, ugi, localDirs, taskSpec, umbilical, 1,
        new HashMap<String, ByteBuffer>(), HashMultimap.<String, String> create(), taskReporter,
        executor, null);

View Full Code Here

Examples of org.apache.tez.runtime.api.impl.TaskSpec

      Set<String> hostsSet = Sets.newHashSet(hosts);
      Set<String> racksSet = Sets.newHashSet(racks);
      locationHint = TaskLocationHint.createTaskLocationHint(hostsSet, racksSet);
    }
    AMSchedulerEventTALaunchRequest lr = new AMSchedulerEventTALaunchRequest(
      taID, capability, new TaskSpec(taID, "dagName", "vertexName", -1,
        ProcessorDescriptor.create("processorClassName"),
      Collections.singletonList(new InputSpec("vertexName",
          InputDescriptor.create("inputClassName"), 1)),
      Collections.singletonList(new OutputSpec("vertexName",
          OutputDescriptor.create("outputClassName"), 1)), null), ta, locationHint,

View Full Code Here

Examples of org.apache.tez.runtime.api.impl.TaskSpec


  TaskSpec createRemoteTaskSpec() {
    Vertex vertex = getVertex();
    ProcessorDescriptor procDesc = vertex.getProcessorDescriptor();
    int taskId = getTaskID().getId();
    return new TaskSpec(getID(),
        vertex.getDAG().getName(),
        vertex.getName(), vertex.getTotalTasks(), procDesc,
        vertex.getInputSpecList(taskId), vertex.getOutputSpecList(taskId), 
        vertex.getGroupInputSpecList(taskId));
  }

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of TaskSpec

This most recent Implementation of TaskSpecDelegate is TaskSpecVRLGLUE3

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

Examples of edu.berkeley.sparrow.daemon.nodemonitor.TaskScheduler.TaskSpec

Examples of org.apache.tez.runtime.api.impl.TaskSpec

Examples of org.apache.tez.runtime.api.impl.TaskSpec

Examples of org.apache.tez.runtime.api.impl.TaskSpec

Examples of org.apache.tez.runtime.api.impl.TaskSpec

Examples of org.apache.tez.runtime.api.impl.TaskSpec