Examples of POCombinerPackage

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage
The package operator that packages the globally rearranged tuples into output format after the combiner stage. It differs from POPackage in that it does not use the index in the NullableTuple to find the bag to put a tuple in. Instead, the inputs are put in a bag corresponding to their offset in the tuple.

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage

                    }
                    bags[mKeyField] = false;
          // Use the POCombiner package in the combine plan
          // as it needs to act differently than the regular
          // package operator.
                    POCombinerPackage combinePack =
                        new POCombinerPackage(pack, bags, keyFieldPositions);
                    mr.combinePlan.add(combinePack);
                    mr.combinePlan.add(cfe);
                    mr.combinePlan.connect(combinePack, cfe);
                    // No need to connect projections in cfe to cp, because
                    // PigCombiner directly attaches output from package to
                    // root of remaining plan.
                    
                    POLocalRearrange mlr = rearrange.clone();
                    fixUpRearrange(mlr);


                    // A specialized local rearrange operator will replace
                    // the normal local rearrange in the map plan. This behaves
                    // like the regular local rearrange in the getNext() 
                    // as far as getting its input and constructing the 
                    // "key" out of the input. It then returns a tuple with
                    // two fields - the key in the first position and the
                    // "value" inside a bag in the second position. This output
                    // format resembles the format out of a Package. This output
                    // will feed to the map foreach which expects this format.
                    // If the key field isn't in the project of the combiner or map foreach,
                    // it is added to the end (This is required so that we can 
                    // set up the inner plan of the new Local Rearrange leaf in the map
                    // and combine plan to contain just the project of the key).
                    patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                    POLocalRearrange clr = rearrange.clone();
                    fixUpRearrange(clr);


                    mr.combinePlan.add(clr);
                    mr.combinePlan.connect(cfe, clr);
                    
                    // Change the package operator in the reduce plan to
                    // be the POCombiner package, as it needs to act
                    // differently than the regular package operator.
                    POCombinerPackage newReducePack =
                        new POCombinerPackage(pack, bags, keyFieldPositions);
                    mr.reducePlan.replace(pack, newReducePack);
                    
                    // the replace() above only changes
                    // the plan and does not change "inputs" to 
                    // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage


                // Use the POCombiner package in the combine plan
                // as it needs to act differently than the regular
                // package operator.
                mr.combinePlan = new PhysicalPlan();
                POCombinerPackage combinePack =
                    new POCombinerPackage(pack, bags);
                mr.combinePlan.add(combinePack);
                mr.combinePlan.add(cfe);
                mr.combinePlan.connect(combinePack, cfe);
                // No need to connect projections in cfe to cp, because
                // PigCombiner directly attaches output from package to
                // root of remaining plan.


                POLocalRearrange mlr = getNewRearrange(rearrange);




                // A specialized local rearrange operator will replace
                // the normal local rearrange in the map plan. This behaves
                // like the regular local rearrange in the getNext() 
                // as far as getting its input and constructing the 
                // "key" out of the input. It then returns a tuple with
                // two fields - the key in the first position and the
                // "value" inside a bag in the second position. This output
                // format resembles the format out of a Package. This output
                // will feed to the map foreach which expects this format.
                // If the key field isn't in the project of the combiner or map foreach,
                // it is added to the end (This is required so that we can 
                // set up the inner plan of the new Local Rearrange leaf in the map
                // and combine plan to contain just the project of the key).
                patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                POLocalRearrange clr = getNewRearrange(rearrange);


                mr.combinePlan.add(clr);
                mr.combinePlan.connect(cfe, clr);


                // Change the package operator in the reduce plan to
                // be the POCombiner package, as it needs to act
                // differently than the regular package operator.
                POCombinerPackage newReducePack =
                    new POCombinerPackage(pack, bags);
                mr.reducePlan.replace(pack, newReducePack);


                // the replace() above only changes
                // the plan and does not change "inputs" to 
                // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage

                    }
                    bags[mKeyField] = false;
          // Use the POCombiner package in the combine plan
          // as it needs to act differently than the regular
          // package operator.
                    POCombinerPackage combinePack =
                        new POCombinerPackage(pack, bags, keyFieldPositions);
                    mr.combinePlan.add(combinePack);
                    mr.combinePlan.add(cfe);
                    mr.combinePlan.connect(combinePack, cfe);
                    // No need to connect projections in cfe to cp, because
                    // PigCombiner directly attaches output from package to
                    // root of remaining plan.
                    
                    POLocalRearrange mlr = rearrange.clone();
                    fixUpRearrange(mlr);


                    // A specialized local rearrange operator will replace
                    // the normal local rearrange in the map plan. This behaves
                    // like the regular local rearrange in the getNext() 
                    // as far as getting its input and constructing the 
                    // "key" out of the input. It then returns a tuple with
                    // two fields - the key in the first position and the
                    // "value" inside a bag in the second position. This output
                    // format resembles the format out of a Package. This output
                    // will feed to the map foreach which expects this format.
                    // If the key field isn't in the project of the combiner or map foreach,
                    // it is added to the end (This is required so that we can 
                    // set up the inner plan of the new Local Rearrange leaf in the map
                    // and combine plan to contain just the project of the key).
                    patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                    POLocalRearrange clr = rearrange.clone();
                    fixUpRearrange(clr);


                    mr.combinePlan.add(clr);
                    mr.combinePlan.connect(cfe, clr);
                    
                    // Change the package operator in the reduce plan to
                    // be the POCombiner package, as it needs to act
                    // differently than the regular package operator.
                    POCombinerPackage newReducePack =
                        new POCombinerPackage(pack, bags, keyFieldPositions);
                    mr.reducePlan.replace(pack, newReducePack);
                    
                    // the replace() above only changes
                    // the plan and does not change "inputs" to 
                    // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage

                    }
                    bags[mKeyField] = false;
          // Use the POCombiner package in the combine plan
          // as it needs to act differently than the regular
          // package operator.
                    POCombinerPackage combinePack =
                        new POCombinerPackage(pack, bags);
                    mr.combinePlan.add(combinePack);
                    mr.combinePlan.add(cfe);
                    mr.combinePlan.connect(combinePack, cfe);
                    // No need to connect projections in cfe to cp, because
                    // PigCombiner directly attaches output from package to
                    // root of remaining plan.
                    
                    POLocalRearrange mlr = rearrange.clone();
                    fixUpRearrange(mlr);


                    // A specialized local rearrange operator will replace
                    // the normal local rearrange in the map plan. This behaves
                    // like the regular local rearrange in the getNext() 
                    // as far as getting its input and constructing the 
                    // "key" out of the input. It then returns a tuple with
                    // two fields - the key in the first position and the
                    // "value" inside a bag in the second position. This output
                    // format resembles the format out of a Package. This output
                    // will feed to the map foreach which expects this format.
                    // If the key field isn't in the project of the combiner or map foreach,
                    // it is added to the end (This is required so that we can 
                    // set up the inner plan of the new Local Rearrange leaf in the map
                    // and combine plan to contain just the project of the key).
                    patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                    POLocalRearrange clr = rearrange.clone();
                    fixUpRearrange(clr);


                    mr.combinePlan.add(clr);
                    mr.combinePlan.connect(cfe, clr);
                    
                    // Change the package operator in the reduce plan to
                    // be the POCombiner package, as it needs to act
                    // differently than the regular package operator.
                    POCombinerPackage newReducePack =
                        new POCombinerPackage(pack, bags);
                    mr.reducePlan.replace(pack, newReducePack);
                    
                    // the replace() above only changes
                    // the plan and does not change "inputs" to 
                    // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage

                    }
                    bags[mKeyField] = false;
          // Use the POCombiner package in the combine plan
          // as it needs to act differently than the regular
          // package operator.
                    POCombinerPackage combinePack =
                        new POCombinerPackage(pack, bags);
                    mr.combinePlan.add(combinePack);
                    mr.combinePlan.add(cfe);
                    mr.combinePlan.connect(combinePack, cfe);
                    // No need to connect projections in cfe to cp, because
                    // PigCombiner directly attaches output from package to
                    // root of remaining plan.
                    
                    POLocalRearrange mlr = rearrange.clone();
                    fixUpRearrange(mlr);


                    // A specialized local rearrange operator will replace
                    // the normal local rearrange in the map plan. This behaves
                    // like the regular local rearrange in the getNext() 
                    // as far as getting its input and constructing the 
                    // "key" out of the input. It then returns a tuple with
                    // two fields - the key in the first position and the
                    // "value" inside a bag in the second position. This output
                    // format resembles the format out of a Package. This output
                    // will feed to the map foreach which expects this format.
                    // If the key field isn't in the project of the combiner or map foreach,
                    // it is added to the end (This is required so that we can 
                    // set up the inner plan of the new Local Rearrange leaf in the map
                    // and combine plan to contain just the project of the key).
                    patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                    POLocalRearrange clr = rearrange.clone();
                    fixUpRearrange(clr);


                    mr.combinePlan.add(clr);
                    mr.combinePlan.connect(cfe, clr);
                    
                    // Change the package operator in the reduce plan to
                    // be the POCombiner package, as it needs to act
                    // differently than the regular package operator.
                    POCombinerPackage newReducePack =
                        new POCombinerPackage(pack, bags);
                    mr.reducePlan.replace(pack, newReducePack);
                    
                    // the replace() above only changes
                    // the plan and does not change "inputs" to 
                    // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage


                // Use the POCombiner package in the combine plan
                // as it needs to act differently than the regular
                // package operator.
                mr.combinePlan = new PhysicalPlan();
                POCombinerPackage combinePack =
                    new POCombinerPackage(pack, bags);
                mr.combinePlan.add(combinePack);
                mr.combinePlan.add(cfe);
                mr.combinePlan.connect(combinePack, cfe);


                // No need to connect projections in cfe to cp, because
                // PigCombiner directly attaches output from package to
                // root of remaining plan.


                POLocalRearrange mlr = getNewRearrange(rearrange);


                POPartialAgg mapAgg = null;
                if(doMapAgg){
                    mapAgg = createPartialAgg(cfe);
                }


                // A specialized local rearrange operator will replace
                // the normal local rearrange in the map plan. This behaves
                // like the regular local rearrange in the getNext() 
                // as far as getting its input and constructing the 
                // "key" out of the input. It then returns a tuple with
                // two fields - the key in the first position and the
                // "value" inside a bag in the second position. This output
                // format resembles the format out of a Package. This output
                // will feed to the map foreach which expects this format.
                // If the key field isn't in the project of the combiner or map foreach,
                // it is added to the end (This is required so that we can 
                // set up the inner plan of the new Local Rearrange leaf in the map
                // and combine plan to contain just the project of the key).
                patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mapAgg, mlr);
                POLocalRearrange clr = getNewRearrange(rearrange);


                mr.combinePlan.add(clr);
                mr.combinePlan.connect(cfe, clr);


                // Change the package operator in the reduce plan to
                // be the POCombiner package, as it needs to act
                // differently than the regular package operator.
                POCombinerPackage newReducePack =
                    new POCombinerPackage(pack, bags);
                mr.reducePlan.replace(pack, newReducePack);


                // the replace() above only changes
                // the plan and does not change "inputs" to 
                // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage


                // Use the POCombiner package in the combine plan
                // as it needs to act differently than the regular
                // package operator.
                mr.combinePlan = new PhysicalPlan();
                POCombinerPackage combinePack =
                    new POCombinerPackage(pack, bags);
                mr.combinePlan.add(combinePack);
                mr.combinePlan.add(cfe);
                mr.combinePlan.connect(combinePack, cfe);
                // No need to connect projections in cfe to cp, because
                // PigCombiner directly attaches output from package to
                // root of remaining plan.


                POLocalRearrange mlr = getNewRearrange(rearrange);




                // A specialized local rearrange operator will replace
                // the normal local rearrange in the map plan. This behaves
                // like the regular local rearrange in the getNext() 
                // as far as getting its input and constructing the 
                // "key" out of the input. It then returns a tuple with
                // two fields - the key in the first position and the
                // "value" inside a bag in the second position. This output
                // format resembles the format out of a Package. This output
                // will feed to the map foreach which expects this format.
                // If the key field isn't in the project of the combiner or map foreach,
                // it is added to the end (This is required so that we can 
                // set up the inner plan of the new Local Rearrange leaf in the map
                // and combine plan to contain just the project of the key).
                patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mlr);
                POLocalRearrange clr = getNewRearrange(rearrange);


                mr.combinePlan.add(clr);
                mr.combinePlan.connect(cfe, clr);


                // Change the package operator in the reduce plan to
                // be the POCombiner package, as it needs to act
                // differently than the regular package operator.
                POCombinerPackage newReducePack =
                    new POCombinerPackage(pack, bags);
                mr.reducePlan.replace(pack, newReducePack);


                // the replace() above only changes
                // the plan and does not change "inputs" to 
                // operators

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage


                // Use the POCombiner package in the combine plan
                // as it needs to act differently than the regular
                // package operator.
                mr.combinePlan = new PhysicalPlan();
                POCombinerPackage combinePack =
                    new POCombinerPackage(pack, bags);
                mr.combinePlan.add(combinePack);
                mr.combinePlan.add(cfe);
                mr.combinePlan.connect(combinePack, cfe);


                // No need to connect projections in cfe to cp, because
                // PigCombiner directly attaches output from package to
                // root of remaining plan.


                POLocalRearrange mlr = getNewRearrange(rearrange);


                POPartialAgg mapAgg = null;
                if(doMapAgg){
                    mapAgg = createPartialAgg(cfe);
                }


                // A specialized local rearrange operator will replace
                // the normal local rearrange in the map plan. This behaves
                // like the regular local rearrange in the getNext() 
                // as far as getting its input and constructing the 
                // "key" out of the input. It then returns a tuple with
                // two fields - the key in the first position and the
                // "value" inside a bag in the second position. This output
                // format resembles the format out of a Package. This output
                // will feed to the map foreach which expects this format.
                // If the key field isn't in the project of the combiner or map foreach,
                // it is added to the end (This is required so that we can 
                // set up the inner plan of the new Local Rearrange leaf in the map
                // and combine plan to contain just the project of the key).
                patchUpMap(mr.mapPlan, getPreCombinerLR(rearrange), mfe, mapAgg, mlr);
                POLocalRearrange clr = getNewRearrange(rearrange);


                mr.combinePlan.add(clr);
                mr.combinePlan.connect(cfe, clr);


                // Change the package operator in the reduce plan to
                // be the POCombiner package, as it needs to act
                // differently than the regular package operator.
                POCombinerPackage newReducePack =
                    new POCombinerPackage(pack, bags);
                mr.reducePlan.replace(pack, newReducePack);


                // the replace() above only changes
                // the plan and does not change "inputs" to 
                // operators

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.