Examples of cascading.flow.planner.BaseFlowStep

cascading.flow.planner.BaseFlowStep
Class FlowStep is an internal representation of a given Job to be executed on a remote cluster. During planning, pipe assemblies are broken down into "steps" and encapsulated in this class.
FlowSteps are submitted in order of dependency. If two or more steps do not share the same dependencies and all can be scheduled simultaneously, the {@link #getSubmitPriority()} value determines the order in whichall steps will be submitted for execution. The default submit priority is 5.
This class is for internal use, there are no stable public methods.


    List steps = getPlatform().getFlowConnector().connect( sources, sinks, pipe ).getFlowSteps();


    assertEquals( "wrong size", 1, steps.size() );


    BaseFlowStep step = (BaseFlowStep) steps.get( 0 );


    assertEquals( "not equal: step.sources.size()", 1, step.getSourceTaps().size() );
    assertNotNull( "null: step.groupBy", step.getGroup() );
    assertNotNull( "null: step.sink", step.getSink() );


    int mapDist = countDistance( step.getElementGraph(), (FlowElement) step.getSourceTaps().iterator().next(), step.getGroup() );
    assertEquals( "not equal: mapDist", 0, mapDist );


    int reduceDist = countDistance( step.getElementGraph(), step.getGroup(), step.getSink() );
    assertEquals( "not equal: reduceDist", 1, reduceDist );
    }

View Full Code Here

    if( testTempReplaced && !flowConnector.getRuleRegistry().hasRule( "CombineAdjacentTapTransformer" ) )
      testTempReplaced = false;


    assertEquals( "not equal: steps.size()", testTempReplaced ? 2 : 3, steps.size() );


    BaseFlowStep step = (BaseFlowStep) steps.get( 0 );


    Scope nextScope = step.getNextScope( step.getGroup() );
    FlowElement operator = step.getNextFlowElement( nextScope );


    assertTrue( "not an Each", operator instanceof Each );


    nextScope = step.getNextScope( operator );
    operator = step.getNextFlowElement( nextScope );


    assertTrue( "not a Each", operator instanceof Each );


    nextScope = step.getNextScope( operator );
    operator = step.getNextFlowElement( nextScope );


    if( testTempReplaced )
      {
      assertEquals( "not proper sink", sinkLeft, operator );
      }

View Full Code Here

    int ordinal = 0;
    int size = vertexSet().size();


    while( iterator.hasNext() )
      {
      BaseFlowStep flowStep = (BaseFlowStep) iterator.next();


      flowStep.setOrdinal( ordinal );
      flowStep.setName( flowPlanner.makeFlowStepName( flowStep, size, ordinal ) );


      ElementGraph stepSubGraph = flowStep.getElementGraph();


      writePlan( ordinal, stepSubGraph, nodeSubGraphsMap.get( stepSubGraph ), pipelineSubGraphsMap );


      ordinal++;
      }

View Full Code Here

0 1

TOP

Related Classes of cascading.flow.planner.BaseFlowStep

cascading.flow.FlowElement

cascading.flow.FlowStepListenersTest

cascading.flow.hadoop.BuildJobsHadoopPlatformTest

cascading.flow.planner.process.FlowStepGraph

cascading.management.CascadingServices

cascading.management.state.ClientState

cascading.operation.debug.BuildDebugPlatformTest

com.twitter.ambrose.cascading.AmbroseCascadingGraphConverter

cascading.flow.FlowException

java.util.Date

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.