List of addTaskLocalFiles() Examples

Examples of addTaskLocalFiles()

org.apache.tez.dag.api.DAG.addTaskLocalFiles()
Set the files etc that must be provided to the tasks of this DAG @param localFiles files that must be available locally for each task. These files may be regular files, archives etc. as specified by the value elements of the map. @return {@link DAG}
org.apache.tez.dag.api.PreWarmVertex.addTaskLocalFiles()
org.apache.tez.dag.api.Vertex.addTaskLocalFiles()
Set the files etc that must be provided to the tasks of this vertex @param localFiles files that must be available locally for each task. These files may be regular files, archives etc. as specified by the value elements of the map. @return this Vertex

Examples of org.apache.tez.dag.api.DAG.addTaskLocalFiles()

    {
    FlowNodeGraph nodeGraph = getFlowNodeGraph();
    Map<FlowNode, Vertex> vertexMap = new HashMap<>();
    DAG dag = DAG.create( getStepDisplayName( initializedConfig.getInt( "cascading.display.id.truncate", Util.ID_LENGTH ) ) );


    dag.addTaskLocalFiles( localResources );


    Iterator<FlowNode> iterator = nodeGraph.getOrderedTopologicalIterator(); // ordering of nodes for consistent remote debugging


    while( iterator.hasNext() )
      {

View Full Code Here

Examples of org.apache.tez.dag.api.PreWarmVertex.addTaskLocalFiles()

        }
        if (doPreWarm) {
          LOG.info("Pre-warming Session");
          PreWarmVertex preWarmVertex = PreWarmVertex.create("PreWarm", preWarmNumContainers, dag
              .getVertex("initialmap").getTaskResource());
          preWarmVertex.addTaskLocalFiles(dag.getVertex("initialmap").getTaskLocalFiles());
          preWarmVertex.setTaskEnvironment(dag.getVertex("initialmap").getTaskEnvironment());
          preWarmVertex.setTaskLaunchCmdOpts(dag.getVertex("initialmap").getTaskLaunchCmdOpts());
          
          tezSession.preWarm(preWarmVertex);
        }

View Full Code Here

Examples of org.apache.tez.dag.api.PreWarmVertex.addTaskLocalFiles()


    if (localResources != null) {
      combinedResources.putAll(localResources);
    }


    prewarmVertex.addTaskLocalFiles(localResources);
    prewarmVertex.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
    prewarmVertex.setTaskEnvironment(getContainerEnvironment(conf, false));
    return prewarmVertex;
  }

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()

                .determineTimestampsAndCacheVisibilities(globalConf);
        // get DelegationToken for each cached file
        ClientDistributedCacheManager.getDelegationTokens(globalConf,
                job.getCredentials());
        MRApps.setupDistributedCache(globalConf, localResources);
        vertex.addTaskLocalFiles(localResources);


        vertex.setTaskLaunchCmdOpts(tezOp.isUseMRMapSettings() ? MRHelpers.getJavaOptsForMRMapper(globalConf)
                : MRHelpers.getJavaOptsForMRReducer(globalConf));


        log.info("For vertex - " + tezOp.getOperatorKey().toString()

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()

    String iReduceStageHistoryText = new String(iROutputStream.toByteArray(), "UTF-8");
    Vertex ivertex = Vertex.create("intermediate_reducer", ProcessorDescriptor.create(
        ReduceProcessor.class.getName())
        .setUserPayload(TezUtils.createUserPayloadFromConf(iReduceStageConf))
        .setHistoryText(iReduceStageHistoryText), 2);
    ivertex.addTaskLocalFiles(commonLocalResources);
    vertices.add(ivertex);


    ByteArrayOutputStream finalReduceOutputStream = new ByteArrayOutputStream(4096);
    finalReduceConf.writeXml(finalReduceOutputStream);
    String finalReduceStageHistoryText = new String(finalReduceOutputStream.toByteArray(), "UTF-8");

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()

    Vertex finalReduceVertex = Vertex.create("finalreduce",
        ProcessorDescriptor.create(
            ReduceProcessor.class.getName())
            .setUserPayload(finalReducePayload)
            .setHistoryText(finalReduceStageHistoryText), 1);
    finalReduceVertex.addTaskLocalFiles(commonLocalResources);
    finalReduceVertex.addDataSink("MROutput",
        MROutputLegacy.createConfigBuilder(finalReduceConf, TextOutputFormat.class, outputPath)
            .build());
    vertices.add(finalReduceVertex);

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()

            intermediateReduceStageConfs[i];
        UserPayload iReduceUserPayload = TezUtils.createUserPayloadFromConf(iconf);
        Vertex ivertex = Vertex.create("ireduce" + (i + 1),
            ProcessorDescriptor.create(ReduceProcessor.class.getName()).
                setUserPayload(iReduceUserPayload), numIReducer);
        ivertex.addTaskLocalFiles(commonLocalResources);
        vertices.add(ivertex);
      }
    }


    Vertex finalReduceVertex = null;

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()

    Vertex finalReduceVertex = null;
    if (numReducer > 0) {
      UserPayload reducePayload = TezUtils.createUserPayloadFromConf(finalReduceConf);
      finalReduceVertex = Vertex.create("reduce", ProcessorDescriptor.create(
          ReduceProcessor.class.getName()).setUserPayload(reducePayload), numReducer);
      finalReduceVertex.addTaskLocalFiles(commonLocalResources);
      finalReduceVertex.addDataSink("MROutput", MROutputLegacy.createConfigBuilder(finalReduceConf,
          NullOutputFormat.class).build());
      vertices.add(finalReduceVertex);
    } else {
      // Map only job

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()


    // Setup stage2 Vertex
    Vertex stage2Vertex = Vertex.create("stage2", ProcessorDescriptor.create(
        FilterByWordOutputProcessor.class.getName()).setUserPayload(TezUtils
        .createUserPayloadFromConf(stage2Conf)), dsd.getNumberOfShards());
    stage2Vertex.addTaskLocalFiles(commonLocalResources);


    // Configure the Output for stage2
    stage2Vertex.addDataSink(
        "MROutput",
        new DataSinkDescriptor(OutputDescriptor.create(MROutput.class.getName())

View Full Code Here

Examples of org.apache.tez.dag.api.Vertex.addTaskLocalFiles()


    // Setup stage2 Vertex
    Vertex stage2Vertex = Vertex.create("stage2", ProcessorDescriptor.create(
        FilterByWordOutputProcessor.class.getName()).setUserPayload(
        TezUtils.createUserPayloadFromConf(stage2Conf)), 1);
    stage2Vertex.addTaskLocalFiles(commonLocalResources);


    // Configure the Output for stage2
    OutputDescriptor od = OutputDescriptor.create(MROutput.class.getName())
        .setUserPayload(TezUtils.createUserPayloadFromConf(stage2Conf));
    OutputCommitterDescriptor ocd =

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.