Examples of cascading.flow.hadoop.planner.HadoopPlanner

cascading.flow.hadoop.planner.HadoopPlanner

Class HadoopPlanner is the core Hadoop MapReduce planner used by default through the {@link cascading.flow.hadoop.HadoopFlowConnector}.

Notes:

Custom JobConf properties
A custom JobConf instance can be passed to this planner by calling {@link #copyJobConf(java.util.Map,org.apache.hadoop.mapred.JobConf)}on a map properties object before constructing a new {@link cascading.flow.hadoop.HadoopFlowConnector}.

A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector. All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting Flow instances.

For example, {@code properties.set("mapred.child.java.opts","-Xmx512m");} would convince Hadoopto spawn all child jvms with a heap of 512MB.

    }


  @Override
  protected FlowPlanner createFlowPlanner()
    {
    return new HadoopPlanner();
    }


    // does this scheme source what it sinks
    if( !successor.getScheme().isSymmetrical() )
      return false;


    HadoopPlanner flowPlanner = (HadoopPlanner) plannerContext.getFlowPlanner();


    URI tempURIScheme = flowPlanner.getDefaultURIScheme( predecessor ); // temp uses default fs
    URI successorURIScheme = flowPlanner.getURIScheme( successor );


    if( !tempURIScheme.equals( successorURIScheme ) )
      return false;


    // safe, both are symmetrical

Examples of cascading.flow.hadoop.planner.HadoopPlanner

Related Classes of cascading.flow.hadoop.planner.HadoopPlanner