Examples of DeltaIteration

eu.stratosphere.api.java.record.operators.DeltaIteration
A DeltaIteration is similar to a {@link BulkIteration}, but maintains state across the individual iteration steps. The state is called the solution set, can be obtained via {@link #getSolutionSet()}, and be accessed by joining (or CoGrouping) with it. The solution set is updated by producing a delta for it, which is merged into the solution set at the end of each iteration step.
The delta iteration must be closed by setting a delta for the solution set ( {@link #setSolutionSetDelta(eu.stratosphere.api.common.operators.Operator)}) and the new workset (the data set that will be fed back, {@link #setNextWorkset(eu.stratosphere.api.common.operators.Operator)}). The DeltaIteration itself represents the result after the iteration has terminated. Delta iterations terminate when the feed back data set (the workset) is empty. In addition, a maximum number of steps is given as a fall back termination guard.
Elements in the solution set are uniquely identified by a key. When merging the solution set delta, contained elements with the same key are replaced.
This class is a subclass of {@code DualInputOperator}. The solution set is considered the first input, the workset is considered the second input.
org.apache.flink.api.java.record.operators.DeltaIteration
s.apache.org/jira/browse/FLINK-1106">FLINK-1106 for more details. A DeltaIteration is similar to a {@link BulkIteration}, but maintains state across the individual iteration steps. The state is called the solution set, can be obtained via {@link #getSolutionSet()}, and be accessed by joining (or CoGrouping) with it. The solution set is updated by producing a delta for it, which is merged into the solution set at the end of each iteration step.
The delta iteration must be closed by setting a delta for the solution set ( {@link #setSolutionSetDelta(org.apache.flink.api.common.operators.Operator)}) and the new workset (the data set that will be fed back, {@link #setNextWorkset(org.apache.flink.api.common.operators.Operator)}). The DeltaIteration itself represents the result after the iteration has terminated. Delta iterations terminate when the feed back data set (the workset) is empty. In addition, a maximum number of steps is given as a fall back termination guard.
Elements in the solution set are uniquely identified by a key. When merging the solution set delta, contained elements with the same key are replaced.
This class is a subclass of {@code DualInputOperator}. The solution set is considered the first input, the workset is considered the second input.

Examples of org.apache.flink.api.java.record.operators.DeltaIteration

    FileDataSource sourceC = new FileDataSource(DummyInputFormat.class, IN_FILE, "Source 3");


    FileDataSink sink1 = new FileDataSink(DummyOutputFormat.class, OUT_FILE, sourceA, "Sink 1");
    FileDataSink sink2 = new FileDataSink(DummyOutputFormat.class, OUT_FILE, sourceC, "Sink 2");


    DeltaIteration iteration = new DeltaIteration(0, "Loop");
    iteration.setInitialSolutionSet(sourceA);
    iteration.setInitialWorkset(sourceB);
    iteration.setMaximumNumberOfIterations(10);


    CrossOperator nextWorkset = CrossOperator.builder(DummyCrossStub.class).name("Next workset").
        input1(iteration.getWorkset()).
        input2(sourceC).
        build();


    JoinOperator solutionSetDelta = JoinOperator.builder(DummyMatchStub.class, LongValue.class,0,0).
        name("Next solution set.").
        input1(nextWorkset).
        input2(iteration.getSolutionSet()).
        build();


    iteration.setNextWorkset(nextWorkset);
    iteration.setSolutionSetDelta(solutionSetDelta);


    FileDataSink sink3 = new FileDataSink(DummyOutputFormat.class, OUT_FILE, iteration, "Sink 3");


    List<FileDataSink> sinks = new ArrayList<FileDataSink>();
    sinks.add(sink1);

View Full Code Here

Examples of org.apache.flink.api.java.record.operators.DeltaIteration

    ReduceOperator reducedSource = ReduceOperator.builder(IdentityReduce.class).
        input(source).
        name("Identity reduce source").
        build();


    DeltaIteration iteration = new DeltaIteration(0,"Loop");
    iteration.setMaximumNumberOfIterations(10);
    iteration.setInitialSolutionSet(source);
    iteration.setInitialWorkset(mappedSource);


    JoinOperator nextWorkset = JoinOperator.builder(DummyNonPreservingMatchStub.class, IntValue.class, 0,0).
        input1(iteration.getWorkset()).
        input2(reducedSource).
        name("Next work set").
        build();


    JoinOperator solutionSetDelta = JoinOperator.builder(DummyNonPreservingMatchStub.class, IntValue.class, 0,
        0).
        input1(iteration.getSolutionSet()).
        input2(nextWorkset).
        name("Solution set delta").
        build();


    iteration.setNextWorkset(nextWorkset);
    iteration.setSolutionSetDelta(solutionSetDelta);


    FileDataSink sink = new FileDataSink(DummyOutputFormat.class, OUT_FILE, iteration, "Iteration sink");
    List<FileDataSink> sinks = new ArrayList<FileDataSink>();
    sinks.add(sink);

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.