Examples of cascading.flow.FlowConnector

cascading.flow.FlowConnector
Class FlowConnector is the base class for all platform planners.
See the {@link FlowDef} class for a fluent way to define a new Flow.
Use the FlowConnector to link source and sink {@link Tap} instances with an assembly of {@link Pipe} instances intoan executable {@link cascading.flow.Flow}.
FlowConnector invokes a planner for the target execution environment.
For executing Flows in local memory against local files, see {@link cascading.flow.local.LocalFlowConnector}.
For Apache Hadoop, see the {@link cascading.flow.hadoop.HadoopFlowConnector}. Or if you have a pre-existing custom Hadoop job to execute, see {@link cascading.flow.hadoop.MapReduceFlow}, which doesn't require a planner.
Note that all {@code connect} methods take a single {@code tail} or an array of {@code tail} Pipe instances. "tail"refers to the last connected Pipe instances in a pipe-assembly. Pipe-assemblies are graphs of object with "heads" and "tails". From a given "tail", all connected heads can be found, but not the reverse. So "tails" must be supplied by the user.
The FlowConnector and the underlying execution framework (Hadoop or local mode) can be configured via a {@link Map} or {@link Properties} instance given to the constructor.
This properties map must be populated before constructing a FlowConnector instance. Many planner specific properties can be set through the {@link FlowConnectorProps} fluent interface.
Some planners have required properties. Hadoop expects {@link AppProps#setApplicationJarPath(java.util.Map,String)} or{@link AppProps#setApplicationJarClass(java.util.Map,Class)} to be set.
Any properties set and passed through the FlowConnector constructor will be global to all Flow instances created through the that FlowConnector instance. Some properties are on the {@link FlowDef} and would only be applicable to theresulting Flow instance.
These properties are used to influence the current planner and are also passed down to the execution framework to override any default values. For example when using the Hadoop planner, the number of reducers or mappers can be set by using platform specific properties.
Custom operations (Functions, Filter, etc) may also retrieve these property values at runtime through calls to {@link cascading.flow.FlowProcess#getProperty(String)} or {@link FlowProcess#getStringProperty(String)}.
Most applications will need to call {@link cascading.property.AppProps#setApplicationJarClass(java.util.Map,Class)} or{@link cascading.property.AppProps#setApplicationJarPath(java.util.Map,String)} so thatthe correct application jar file is passed through to all child processes. The Class or path must reference the custom application jar, not a Cascading library class or jar. The easiest thing to do is give setApplicationJarClass the Class with your static main function and let Cascading figure out which jar to use.
Note that Map is compatible with the {@link Properties} class, so properties can be loaded atruntime from a configuration file.
By default, all {@link cascading.operation.Assertion}s are planned into the resulting Flow instance. This can be changed for a given Flow by calling {@link FlowDef#setAssertionLevel(cascading.operation.AssertionLevel)} or globallyvia {@link FlowConnectorProps#setAssertionLevel(cascading.operation.AssertionLevel)}.
Also by default, all {@link cascading.operation.Debug}s are planned into the resulting Flow instance. This can be changed for a given flow by calling {@link FlowDef#setDebugLevel(cascading.operation.DebugLevel)} or globally via{@link FlowConnectorProps#setDebugLevel(cascading.operation.DebugLevel)}.
As of version 3.0, custom {@link cascading.flow.planner.rule.RuleRegistry} instances can be provided to customizea given planner. @see cascading.flow.local.LocalFlowConnector @see cascading.flow.hadoop.HadoopFlowConnector

    // set the current job jar
    Properties properties = new Properties();
    FlowConnector.setApplicationJarClass( properties, Main.class );


    // connect the assembly to the SOURCE and SINK taps
    Flow parsedLogFlow = new FlowConnector( properties ).connect( logTap, remoteLogTap, importPipe );


    // optionally print out the parsedLogFlow to a DOT file for import into a graphics package
    // parsedLogFlow.writeDOT( "logparser.dot" );


    // start execution of the flow (either locally or on the cluster

View Full Code Here

        properties.setProperty("mapred.create.symlink", "yes");
      }
    }


    FlowConnector.setApplicationJarClass(properties, Main.class);
    FlowConnector flowConnector = new FlowConnector(properties);
    Flow flow = flowConnector.connect(sources, sinks, tails);
    if ("hadoop".equals(runningMode)) {
      try {
        flow.addListener(tempDir);
      } catch (Exception e) {
        e.printStackTrace();

View Full Code Here

    Tap sinkSecond = getPlatform().getTabDelimitedFile( new Fields( "ip" ), getOutputPath( "chainedtaps/second" ), SinkMode.REPLACE );
    Tap sinkThird = getPlatform().getTabDelimitedFile( new Fields( "ip" ), getOutputPath( "chainedtaps/third" ), SinkMode.REPLACE );


    Map<String, Tap> sinks = Cascades.tapsMap( new String[]{"first", "second",
                                                            "third"}, Tap.taps( sinkFirst, sinkSecond, sinkThird ) );
    FlowConnector flowConnector = getPlatform().getFlowConnector();


    Flow flow = flowConnector.connect( source, sinks, pipe );


    if( getPlatform().isMapReduce() )
      assertEquals( "wrong number of steps", 3, flow.getFlowSteps().size() );


    flow.complete();

View Full Code Here

    Pipe splice2 = new CoGroup( splice1, new Fields( "num1" ), pipeNum102, new Fields( "num" ), new Fields( "num1", "num2", "num3" ) );


    Properties properties = new Properties();
    FlowConnectorProps.setIntermediateSchemeClass( properties, TextLine.class );


    FlowConnector flowConnector = getPlatform().getFlowConnector( properties );


    Flow flow = flowConnector.connect( sources, sink, splice2 );


    assertEquals( "not equal: steps.size()", 2, flow.getFlowSteps().size() );
    }

View Full Code Here


    Pipe splice1 = new CoGroup( pipeNum20, new Fields( "num" ), pipeNum101, new Fields( "num" ), new Fields( "num1", "num2" ) );


    Pipe splice2 = new CoGroup( splice1, new Fields( "num9" ), pipeNum102, new Fields( "num" ), new Fields( "num1", "num2", "num3" ) );


    FlowConnector flowConnector = getPlatform().getFlowConnector();


    try
      {
      Flow flow = flowConnector.connect( sources, sink, splice2 );
      fail( "did not fail on bad field" );
      }
    catch( Exception exception )
      {
      // ignore

View Full Code Here

    right = new Each( right, new Fields( "num" ), new RegexFilter( ".*" ) );


    if( !before )
      right = new Pipe( "right", right );


    FlowConnector flowConnector = getPlatform().getFlowConnector();


    Flow flow = flowConnector.connect( "splitmiddle", sources, sinks, left, right );


    List<FlowStep> steps = flow.getFlowSteps();


    // if the optimization isn't in place, don't test for it
    if( testTempReplaced && !flowConnector.getRuleRegistry().hasRule( "CombineAdjacentTapTransformer" ) )
      testTempReplaced = false;


    assertEquals( "not equal: steps.size()", testTempReplaced ? 2 : 3, steps.size() );


    BaseFlowStep step = (BaseFlowStep) steps.get( 0 );

View Full Code Here

      .setName( "sql flow" )
      .addSource( "example.employee", empTap )
      .addSource( "example.sales_fact_1997", salesTap )
      .addSink( "results", resultsTap );


    FlowConnector flowConnector = new LocalFlowConnector();


    SQLPlanner sqlPlanner = new SQLPlanner()
      .setSql( statement );


    flowDef.addAssemblyPlanner( sqlPlanner );


    Flow flow = flowConnector.connect( flowDef );


    flow.complete();


    TupleEntryIterator iterator = resultsTap.openForRead( flow.getFlowProcess() );

View Full Code Here

0 1 2 3

TOP

Related Classes of cascading.flow.FlowConnector

bixo.examples.crawl.DemoCrawlWorkflow

bixo.examples.crawl.LatestUrlDatumBufferTest

bixo.examples.crawl.UrlImporter

bixo.examples.webmining.DemoWebMiningWorkflow

bixo.fetcher.FetcherTest

bixo.pipes.AbstractFetchPipeTest

bixo.pipes.ParsePipeTest

bixo.pipes.RunFakeFetchPipe

bixo.tools.LengthenUrlsTool

cascading.flow.hadoop.BuildJobsHadoopPlatformTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.