Examples of cascading.flow.FlowProcess

cascading.flow.FlowProcess
FlowProcess implementations provide a call-back interface into the current computing system. Each {@link cascading.operation.Operation} is given a reference to a particular implementation, allowing itto get configuration properties, send a "keep alive" ping, or to set a counter value.
Depending on the underlying system, FlowProcess instances are not continuous across all operations in a {@link Flow}. Thus, a call to {@link #increment(Enum,long)} may start incrementing from zero if the operation making the callbelongs to a subsequent 'job' or 'step' from any previous operations calling increment.
A FlowProcess is roughly a child of {@link FlowSession}. FlowSession is roughly one to one with a particular {@link Flow}. And every FlowSession will have one or more FlowProcesses. @see FlowSession

        SimpleHttpFetcher fetcher = new SimpleHttpFetcher(maxThreads, ConfigUtils.BIXO_TEST_AGENT);
        BaseScoreGenerator scorer = new FixedScoreGenerator(1.0);
        BaseRobotsParser parser = new SimpleRobotRulesParser();
        FilterAndScoreByUrlAndRobots op = new FilterAndScoreByUrlAndRobots(fetcher, parser, scorer);
        
        FlowProcess fp = Mockito.mock(NullFlowProcess.class);
        
        OperationCall<NullContext> oc = Mockito.mock(OperationCall.class);
        BufferCall<NullContext> bc = Mockito.mock(BufferCall.class);
        
        TupleEntryCollector collector = Mockito.mock(TupleEntryCollector.class);

View Full Code Here

        SimpleHttpFetcher fetcher = new SimpleHttpFetcher(maxThreads, ConfigUtils.BIXO_TEST_AGENT);
        BaseScoreGenerator scorer = new FixedScoreGenerator(1.0);
        BaseRobotsParser parser = new SimpleRobotRulesParser();
        FilterAndScoreByUrlAndRobots op = new FilterAndScoreByUrlAndRobots(fetcher, parser, scorer);
        
        FlowProcess fp = Mockito.mock(NullFlowProcess.class);
        
        OperationCall<NullContext> oc = Mockito.mock(OperationCall.class);
        BufferCall<NullContext> bc = Mockito.mock(BufferCall.class);
        
        TupleEntryCollector collector = Mockito.mock(TupleEntryCollector.class);

View Full Code Here


    int count = 0;


    for( int i = 0; i < tasks; i++ )
      {
      FlowProcess process = new TestFlowProcess( tasks, i );


      filter.prepare( process, operationCall );


      operationCall.setArguments( getEntry( new Tuple( 1 ) ) );

View Full Code Here

      else
        {
        conf = (TezConfiguration) inputConfigMap.get( FlowElements.id( flowElement ) );
        }


      FlowProcess flowProcess = conf == null ? tezProcess : new Hadoop2TezFlowProcess( tezProcess, conf );


      handleHead( flowElement, flowProcess );
      }
    }

View Full Code Here


  @Test
  public void testHadoopConf() {
    CombinedSequenceFile csfScheme = new CombinedSequenceFile(Fields.ALL);
    JobConf conf = new JobConf();
    FlowProcess fp = new HadoopFlowProcess();
    Tap<JobConf, RecordReader, OutputCollector> tap =
        new TempHfs(conf, "test", CombinedSequenceFile.class, false);


    csfScheme.sourceConfInit(fp, tap, conf);

View Full Code Here

    PlatformBroker platformBroker = getPlatformBroker();
    Properties properties = platformBroker.getProperties();


    Optiq.writeSQLPlan( properties, Misc.createUniqueName(), getVolcanoPlanner() );


    FlowProcess flowProcess = platformBroker.getFlowProcess();
    SchemaCatalogManager schemaCatalog = platformBroker.getCatalogManager();


    Tap tap = schemaCatalog.createTapFor( getTableDef(), SinkMode.KEEP );
    int size = tap.getSourceFields().size();

View Full Code Here

    ClassLoader jarLoader = ClassLoaderUtil.getJarClassLoader( platformBroker, flowFactory );


    if( jarLoader != null )
      Thread.currentThread().setContextClassLoader( jarLoader );


    FlowProcess flowProcess = platformBroker.getFlowProcess();
    SchemaCatalogManager schemaCatalog = platformBroker.getCatalogManager();
    Map<String, TupleEntryCollector> cache = platformBroker.getCollectorCache();


    TupleEntryCollector collector;

View Full Code Here

TOP

Related Classes of cascading.flow.FlowProcess

bixo.operations.FilterAndScoreByUrlAndRobotsTest

cascading.flow.tez.stream.graph.Hadoop2TezStreamGraph

cascading.lingual.optiq.enumerable.CascadingTapEnumerable

cascading.lingual.optiq.enumerable.CascadingValueInsertEnumerable

cascading.operation.filter.LimitFilterTest

com.twitter.elephantbird.cascading2.scheme.TestCombinedSequenceFile

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.