Examples of MapOperator

eu.stratosphere.api.java.record.operators.MapOperator
MapOperator that applies a {@link MapFunction} to each record independently. @see MapFunction
org.apache.flink.api.java.record.operators.MapOperator
s.apache.org/jira/browse/FLINK-1106">FLINK-1106 for more details. MapOperator that applies a {@link MapFunction} to each record independently. @see MapFunction
org.apache.hadoop.hive.ql.exec.MapOperator
Map operator. This triggers overall map side processing. This is a little different from regular operators in that it starts off by processing a Writable data structure from a Table (instead of a Hive Object).

Examples of org.apache.flink.api.java.record.operators.MapOperator

      int numSubTasks, String verticesInput, String edgeInput, String output, int maxIterations)
  {
    // create DataSourceContract for the vertices
    FileDataSource initialVertices = new FileDataSource(new CsvInputFormat(' ', LongValue.class), verticesInput, "Vertices");
    
    MapOperator verticesWithId = MapOperator.builder(DuplicateLongMap.class).input(initialVertices).name("Assign Vertex Ids").build();
    
    DeltaIteration iteration = new DeltaIteration(0, "Connected Components Iteration");
    iteration.setInitialSolutionSet(verticesWithId);
    iteration.setInitialWorkset(verticesWithId);
    iteration.setMaximumNumberOfIterations(maxIterations);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    // Example with Wrapper Converter
    HadoopDataSource<LongWritable,Text> sourceHadoopType = new HadoopDataSource<LongWritable, Text>(
        new TextInputFormat(), new JobConf(), "Input Lines", new WritableWrapperConverter<LongWritable, Text>());
    TextInputFormat.addInputPath(source.getJobConf(), new Path(dataInput));
    
    MapOperator mapper = MapOperator.builder(new TokenizeLine())
      .input(source)
      .name("Tokenize Lines")
      .build();
    ReduceOperator reducer = ReduceOperator.builder(CountWords.class, StringValue.class, 0)
      .input(mapper)

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    
    // init data source 
    FileDataSource input = new FileDataSource(new ContractITCaseInputFormat(), inputPath);


    // init failing map task
    MapOperator testMapper = MapOperator.builder(FailingMapper.class).build();


    // init data sink
    FileDataSink output = new FileDataSink(new ContractITCaseOutputFormat(), resultPath);


    // compose failing program
    output.setInput(testMapper);
    testMapper.setInput(input);


    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(DOP);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    
    // init data source 
    FileDataSource input = new FileDataSource(new ContractITCaseInputFormat(), inputPath);


    // init (working) map task
    MapOperator testMapper = MapOperator.builder(TestMapper.class).build();


    // init data sink
    FileDataSink output = new FileDataSink(new ContractITCaseOutputFormat(), resultPath);


    // compose working program
    output.setInput(testMapper);
    testMapper.setInput(input);


    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(4);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    HadoopDataSource<LongWritable, Text> source = new HadoopDataSource<LongWritable, Text>(
        new TextInputFormat(), new JobConf(), "Input Lines");
    TextInputFormat.addInputPath(source.getJobConf(), new Path(dataInput));




    MapOperator mapper = MapOperator.builder(new TokenizeLine())
        .input(source)
        .name("Tokenize Lines")
        .build();
    ReduceOperator reducer = ReduceOperator.builder(CountWords.class, StringValue.class, 0)
        .input(mapper)

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

  public static Plan getPlan(int numSubTasks, String verticesInput, String edgeInput, String output, int maxIterations) {


    // data source for initial vertices
    FileDataSource initialVertices = new FileDataSource(new CsvInputFormat(' ', LongValue.class), verticesInput, "Vertices");
    
    MapOperator verticesWithId = MapOperator.builder(DuplicateLongMap.class).input(initialVertices).name("Assign Vertex Ids").build();
    
    DeltaIteration iteration = new DeltaIteration(0, "Connected Components Iteration");
    iteration.setInitialSolutionSet(verticesWithId);
    iteration.setInitialWorkset(verticesWithId);
    iteration.setMaximumNumberOfIterations(maxIterations);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    CsvInputFormat visitsFormat = new CsvInputFormat('|', null, StringValue.class, StringValue.class);
    FileDataSource visits = new FileDataSource(visitsFormat, visitsInput, "Visits input:q");


    // Create MapOperator for filtering the entries from the documents
    // relation
    MapOperator filterDocs = MapOperator.builder(new FilterDocs())
      .input(docs)
      .name("Filter Docs")
      .build();
    filterDocs.getCompilerHints().setFilterFactor(0.15f);


    // Create MapOperator for filtering the entries from the ranks relation
    MapOperator filterRanks = MapOperator.builder(new FilterRanks())
      .input(ranks)
      .name("Filter Ranks")
      .build();
    filterRanks.getCompilerHints().setFilterFactor(0.25f);


    // Create MapOperator for filtering the entries from the visits relation
    MapOperator filterVisits = MapOperator.builder(new FilterVisits())
      .input(visits)
      .name("Filter Visits")
      .build();
    filterVisits.getCompilerHints().setFilterFactor(0.2f);


    // Create JoinOperator to join the filtered documents and ranks
    // relation
    JoinOperator joinDocsRanks = JoinOperator.builder(new JoinDocRanks(), StringValue.class, 0, 0)
      .input1(filterDocs)

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

    


    FileDataSource edges = new FileDataSource(new EdgeInputFormat(), edgeInput, "Input Edges");
    edges.setParameter(EdgeInputFormat.ID_DELIMITER_CHAR, delimiter);
    
    MapOperator projectEdge = MapOperator.builder(new ProjectEdge())
      .input(edges).name("Project Edge").build();
    
    ReduceOperator edgeCounter = ReduceOperator.builder(new CountEdges(), IntValue.class, 0)
      .input(projectEdge).name("Count Edges for Vertex").build();

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

        .name("Compute sum (Reduce)")
        .build();
    
    iteration.setNextPartialSolution(sumReduce);
    
    MapOperator terminationMapper = MapOperator.builder(new TerminationMapper())
        .input(iteration.getPartialSolution())
        .name("Compute termination criterion (Map)")
        .build();
    
    iteration.setTerminationCriterion(terminationMapper);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.MapOperator

     */
    public Plan getPlan(String inputPath, String outputPath, int numSubtasks) {


      FileDataSource source = new FileDataSource(TextInputFormat.class, inputPath, "First Input");


      MapOperator wordsFirstInput = MapOperator.builder(TokenizeLine.class)
        .input(source)
        .name("Words (First Input)")
        .build();


      MapOperator wordsSecondInput = MapOperator.builder(TokenizeLine.class)
        .input(source)
        .name("Words (Second Input)")
        .build();


      @SuppressWarnings("unchecked")

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.