Examples of Reducer

com.hazelcast.mapreduce.Reducer
The abstract Reducer class is used to build reducers for the {@link Job}.
Reducers may be distributed inside of the cluster but there is always only one Reducer per key.
Reducers are called in a threadsafe way so internal locking is not required.

Due to the fact that there is only one Reducer per key mapped values needs to be transmitted to one of the cluster nodes. To reduce the traffic costs between the nodes a {@link Combiner} implementation can be added to the call which runs alongsidethe mapper to pre-reduce mapped values into intermediate results.

A simple Reducer implementation could look like that sum-function implementation:
```
 public class SumReducer implements Reducer<String, Integer, Integer> { private int sum = 0; public void reduce( String key, Integer value ) { sum += value; } public Integer finalizeReduce() { return sum; } } 
```
@param < KeyIn> key type of the resulting keys @param < ValueIn> value type of the incoming values @param < ValueOut> value type of the reduced values @since 3.2
loop.Reducer
Takes an AST generated by the parser and strips it of unnecessary elements. Also optimizes the AST to be in its most compact form. This is sometimes referred to as reducing the "parse tree" to an AST (abstract syntax tree). Nothing done in this phase affects the semantics of the program.
org.apache.hadoop.hive.contrib.mr.Reducer
Simple reducer interface.
org.apache.hadoop.mapred.Reducer
ess int noValues = 0; while (values.hasNext()) { V value = values.next(); // Increment the no. of values for this key ++noValues; // Process the <key, value> pair (assume this takes a while) // ... // ... // Let the framework know that we are alive, and kicking! if ((noValues%10) == 0) { reporter.progress(); } // Process some more // ... // ... // Output the <key, value> output.collect(key, value); } // Increment the no. of <key, list of values> pairs processed ++noKeys; // Increment counters reporter.incrCounter(NUM_RECORDS, 1); // Every 100 keys update application-level status if ((noKeys%100) == 0) { reporter.setStatus(reduceTaskId + " processed " + noKeys); } } }
@see Mapper @see Partitioner @see Reporter @see MapReduceBase
org.apache.hadoop.mapreduce.Reducer
Reduces a set of intermediate values which share a key to a smaller set of values.
Reducer implementations can access the {@link Configuration} for the job via the {@link JobContext#getConfiguration()} method.
Reducer has 3 primary phases:
1. Shuffle
  
  The Reducer copies the sorted output from each {@link Mapper} using HTTP across the network.
2. Sort
  
  The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key).
  
  The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.
  
  SecondarySort
  
  To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce.The grouping comparator is specified via {@link Job#setGroupingComparatorClass(Class)}. The sort order is controlled by {@link Job#setSortComparatorClass(Class)}.
  For example, say that you want to find duplicate web pages and tag them all with the url of the "best" known example. You would set up the job like:
  - Map Input Key: url
  - Map Input Value: document
  - Map Output Key: document checksum, url pagerank
  - Map Output Value: url
  - Partitioner: by checksum
  - OutputKeyComparator: by checksum and then decreasing pagerank
  - OutputValueGroupingComparator: by checksum
3. Reduce
  
  In this phase the {@link #reduce(Object,Iterable,Context)}method is called for each <key, (collection of values)> in the sorted inputs.
  
  The output of the reduce task is typically written to a {@link RecordWriter} via {@link Context#write(Object,Object)}.
The output of the Reducer is not re-sorted.

Example:
```
 public class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Key key, Iterable values,  Context context) throws IOException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.collect(key, result); } } 
```
@see Mapper @see Partitioner
rationals.transformations.Reducer

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getCombinerKeyGroupingComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Reducer

  public void testUseOfWritableRegisteredComparator() throws IOException {
    
    // this test should use the comparator registered inside TestWritable
    // to output the keys in reverse order
    MapReduceDriver<TestWritable,Text,TestWritable,Text,TestWritable,Text> driver 
      = MapReduceDriver.newMapReduceDriver(new Mapper(), new Reducer());
    
    driver.withInput(new TestWritable("A1"), new Text("A1"))
      .withInput(new TestWritable("A2"), new Text("A2"))
      .withInput(new TestWritable("A3"), new Text("A3"))
      .withKeyGroupingComparator(new TestWritable.SingleGroupComparator())

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Reducer


  @Test
  public void testClosedFormMapReduce() throws IOException {
    
    MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver 
      = MapReduceDriver.newMapReduceDriver(new StatefulMapper(), new Reducer());
    
    mapReduceDriver.addInput(new LongWritable(1L), new Text("hello"));
    mapReduceDriver.addInput(new LongWritable(2L), new Text("schmo"));
    mapReduceDriver.withOutput(new Text("SomeKey"), new IntWritable(2));
    mapReduceDriver.runTest();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Reducer

  /**
   * Tests that method context.getNumReduceTasks() correctly returns value from configuration
   * */
  @Test
  public void testContext_getNumReduceTasks() throws Exception {
    Reducer reducer = new TestReducer();
    ReduceDriver driver = ReduceDriver.newReduceDriver(reducer);
    driver.getConfiguration().setInt("mapred.reduce.tasks", 10);


    List<Text> values = new ArrayList();
    values.add(new Text("bb"));

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of Reducer

Shuffle

Sort

SecondarySort

Reduce

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer

Examples of org.apache.hadoop.mapreduce.Reducer