Examples of org.apache.hadoop.hive.ql.ppd.PredicatePushDown

org.apache.hadoop.hive.ql.ppd.PredicatePushDown
Implements predicate pushdown. Predicate pushdown is a term borrowed from relational databases even though for Hive it is predicate pushup. The basic idea is to process expressions as early in the plan as possible. The default plan generation adds filters where they are seen but in some instances some of the filter expressions can be pushed nearer to the operator that sees this particular data for the first time. e.g. select a.*, b.* from a join b on (a.col1 = b.col1) where a.col1 > 20 and b.col2 > 40 For the above query, the predicates (a.col1 > 20) and (b.col2 > 40), without predicate pushdown, would be evaluated after the join processing has been done. Suppose the two predicates filter out most of the rows from a and b, the join is unnecessarily processing these rows. With predicate pushdown, these two predicates will be processed before the join. Predicate pushdown is enabled by setting hive.optimize.ppd to true. It is disable by default. The high-level algorithm is describe here - An operator is processed after all its children have been processed - An operator processes its own predicates and then merges (conjunction) with the processed predicates of its children. In case of multiple children, there are combined using disjunction (OR). - A predicate expression is processed for an operator using the following steps - If the expr is a constant then it is a candidate for predicate pushdown - If the expr is a col reference then it is a candidate and its alias is noted - If the expr is an index and both the array and index expr are treated as children - If the all child expr are candidates for pushdown and all of the expression reference only one alias from the operator's RowResolver then the current expression is also a candidate One key thing to note is that some operators (Select, ReduceSink, GroupBy, Join etc) change the columns as data flows through them. In such cases the column references are replaced by the corresponding expression in the input data.

    transformations = new ArrayList<Transform>();
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCP)) {
      transformations.add(new ColumnPruner());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
    }
    
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTGROUPBY)) {
      transformations.add(new GroupByOptimizer());

View Full Code Here

    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCP)) {
      transformations.add(new ColumnPruner());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicateTransitivePropagate());
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTLISTBUCKETING)) {
        /* Add list bucketing pruner. */
        transformations.add(new ListBucketingPruner());

View Full Code Here

    transformations = new ArrayList<Transform>();
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCP)) {
      transformations.add(new ColumnPruner());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicatePushDown());
      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPR)) {
        transformations.add(new PartitionPruner());
      }
    }
    transformations.add(new UnionProcessor());

View Full Code Here

    transformations.add(new Generator());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCP)) {
      transformations.add(new ColumnPruner());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTGBYUSINGINDEX)) {
      transformations.add(new RewriteGBUsingIndex());

View Full Code Here

    transformations = new ArrayList<Transform>();
    // Add the transformation that computes the lineage information.
    transformations.add(new Generator());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicateTransitivePropagate());
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTLISTBUCKETING)) {
        /* Add list bucketing pruner. */
        transformations.add(new ListBucketingPruner());

View Full Code Here

    transformations.add(new PredicateTransitivePropagate());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCONSTANTPROPAGATION)) {
      transformations.add(new ConstantPropagate());
    }
      transformations.add(new SyntheticJoinPredicate());
      transformations.add(new PredicatePushDown());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCONSTANTPROPAGATION)) {
      // We run constant propagation twice because after predicate pushdown, filter expressions
      // are combined and may become eligible for reduction (like is not null filter).
        transformations.add(new ConstantPropagate());

View Full Code Here

    transformations = new ArrayList<Transform>();
    // Add the transformation that computes the lineage information.
    transformations.add(new Generator());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicateTransitivePropagate());
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTLISTBUCKETING)) {
        /* Add list bucketing pruner. */
        transformations.add(new ListBucketingPruner());

View Full Code Here

    transformations = new ArrayList<Transform>();
    // Add the transformation that computes the lineage information.
    transformations.add(new Generator());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicateTransitivePropagate());
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTLISTBUCKETING)) {
        /* Add list bucketing pruner. */
        transformations.add(new ListBucketingPruner());

View Full Code Here

    transformations.add(new Generator());
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTCP)) {
      transformations.add(new ColumnPruner());
    }
    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
      transformations.add(new PredicatePushDown());
      transformations.add(new PartitionPruner());
      transformations.add(new PartitionConditionRemover());
    }


    if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTGROUPBY)) {

View Full Code Here

TOP

Related Classes of org.apache.hadoop.hive.ql.ppd.PredicatePushDown

org.apache.hadoop.hive.ql.lib.DefaultGraphWalker

org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher

org.apache.hadoop.hive.ql.lib.Dispatcher

org.apache.hadoop.hive.ql.lib.GraphWalker

org.apache.hadoop.hive.ql.lib.RuleRegExp

org.apache.hadoop.hive.ql.optimizer.Optimizer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.