Examples of Reducer

• com.hazelcast.mapreduce.Reducer

  The abstract Reducer class is used to build reducers for the {@link Job}.
  Reducers may be distributed inside of the cluster but there is always only one Reducer per key.
  Reducers are called in a threadsafe way so internal locking is not required.

  Due to the fact that there is only one Reducer per key mapped values needs to be transmitted to one of the cluster nodes. To reduce the traffic costs between the nodes a {@link Combiner} implementation can be added to the call which runs alongsidethe mapper to pre-reduce mapped values into intermediate results.

  A simple Reducer implementation could look like that sum-function implementation:

   public class SumReducer implements Reducer<String, Integer, Integer> { private int sum = 0; public void reduce( String key, Integer value ) { sum += value; } public Integer finalizeReduce() { return sum; } } 

  @param < KeyIn> key type of the resulting keys @param < ValueIn> value type of the incoming values @param < ValueOut> value type of the reduced values @since 3.2
• loop.Reducer
  Takes an AST generated by the parser and strips it of unnecessary elements. Also optimizes the AST to be in its most compact form. This is sometimes referred to as reducing the "parse tree" to an AST (abstract syntax tree). Nothing done in this phase affects the semantics of the program.
• org.apache.hadoop.hive.contrib.mr.Reducer
  Simple reducer interface.
• org.apache.hadoop.mapred.Reducer
  ess int noValues = 0; while (values.hasNext()) { V value = values.next(); // Increment the no. of values for this key ++noValues; // Process the <key, value> pair (assume this takes a while) // ... // ... // Let the framework know that we are alive, and kicking! if ((noValues%10) == 0) { reporter.progress(); } // Process some more // ... // ... // Output the <key, value> output.collect(key, value); } // Increment the no. of <key, list of values> pairs processed ++noKeys; // Increment counters reporter.incrCounter(NUM_RECORDS, 1); // Every 100 keys update application-level status if ((noKeys%100) == 0) { reporter.setStatus(reduceTaskId + " processed " + noKeys); } } }

  @see Mapper @see Partitioner @see Reporter @see MapReduceBase
• org.apache.hadoop.mapreduce.Reducer
  Reduces a set of intermediate values which share a key to a smaller set of values.

  Reducer implementations can access the {@link Configuration} for the job via the {@link JobContext#getConfiguration()} method.

  Reducer has 3 primary phases:

  1. Shuffle

     The Reducer copies the sorted output from each {@link Mapper} using HTTP across the network.

  2. Sort

     The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key).

     The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.

     SecondarySort

     To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce.The grouping comparator is specified via {@link Job#setGroupingComparatorClass(Class)}. The sort order is controlled by {@link Job#setSortComparatorClass(Class)}.

     For example, say that you want to find duplicate web pages and tag them all with the url of the "best" known example. You would set up the job like:
     • Map Input Key: url
     • Map Input Value: document
     • Map Output Key: document checksum, url pagerank
     • Map Output Value: url
     • Partitioner: by checksum
     • OutputKeyComparator: by checksum and then decreasing pagerank
     • OutputValueGroupingComparator: by checksum

  3. Reduce

     In this phase the {@link #reduce(Object,Iterable,Context)}method is called for each <key, (collection of values)> in the sorted inputs.

     The output of the reduce task is typically written to a {@link RecordWriter} via {@link Context#write(Object,Object)}.

  The output of the Reducer is not re-sorted.

  Example:

   public class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Key key, Iterable values,  Context context) throws IOException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.collect(key, result); } } 

  @see Mapper @see Partitioner
• rationals.transformations.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer

Examples of rationals.transformations.Reducer