Examples of com.datasalt.pangool.tuplemr.TupleMRBuilder$Output

net.sf.laja.parser.engine2.Output
TupleMRBuilder creates Tuple-based Map-Reduce jobs.
One of the key concepts of Tuple-based Map-Reduce is that Hadoop Key-Value pairs are no longer used.Instead,they are replaced by tuples.
Tuples(see {@link ITuple}) are just an ordered list of elements whose types are defined in a {@link Schema}.TupleMRBuilder contains several methods to define how grouping and sorting among tuples will be performed, avoiding the complex task of defining custom binary {@link SortComparator} ,{@link GroupComparator} and {@link TupleHashPartitioner} implementations.
A Tuple-based Map-Red job, in its simplest form, requires to define :
- Intermediate schemas:
  An schema specifies the name and types of a Tuple's fields. Several schemas can be defined in order to perform joins among different input data. It's mandatory to specify ,at least,one schema using {@link #addIntermediateSchema(Schema)}
- Group-by fields:
  Needed to specify how the tuples will be grouped. Several tuples with the same group-by fields will be groupped and reduced together in the Reduce phase.
- Tuple-based Mapper:
  The job needs to specify a {@link TupleMapper} instance,the Tuple-basedimplementation of Hadoop's {@link Mapper}. Unlike Hadoop's Mappers, Tuple-based mappers are configured using stateful serializable instances and not static class definitions.
- Tuple-based Reducer: Similar to mapper instances,the job needs to specify a {@link TupleReducer} instance,the Tuple-based implementation ofHadoop's {@link Reducer}.
@see ITuple @see Schema @see TupleMapper @see TupleReducer


      Schema schema = new Schema("sch", Fields.parse(pangoolSchema));
      Path inputP = new Path(inputPath);


      // Use Pangool API - parse CSV, etc
      TupleMRBuilder builder = new TupleMRBuilder(conf);
      TupleTextInputFormat parsingInputFormat = new TupleTextInputFormat(schema, skipHeading, false,
          separator.charAt(0), quotes.charAt(0), escape.charAt(0), FieldSelector.NONE, null);
      TupleTextOutputFormat outputFormat = new TupleTextOutputFormat(schema, false, separator.charAt(0),
          quotes.charAt(0), escape.charAt(0));


      builder.addIntermediateSchema(schema);
      builder.addInput(inputP, parsingInputFormat, new IdentityTupleMapper());
      builder.setGroupByFields(groupBy);
      builder.setOutput(outP, outputFormat, ITuple.class, NullWritable.class);
      builder.setTupleReducer(new IdentityTupleReducer());
      builder.setJarByClass(this.getClass());
      
      builder.createJob().waitForCompletion(true);
    }


    return 1;
  }

View Full Code Here

    for (Iterator i = list.iterator(); i.hasNext();) {
      source.setBookmark(bookmark);


      ElementOutputList.ElementOutput elementOutput = (ElementOutputList.ElementOutput)i.next();
      Element element = elementOutput.getElement();
      Output output = elementOutput.getOutput();
      
      boolean elementNot = elementOutput.isNot();


      if (this.caseSensitive != null) {
        caseSensitive = this.caseSensitive.booleanValue();
      }
      if (elementOutput.isCaseSensitive() != null) {
        caseSensitive = elementOutput.isCaseSensitive().booleanValue();
      }


      boolean match = false;
      if (output.receive()) {
        if (receive) {
          match = recorder.playMatch(element, source.getBookmark());
          source.setBookmark(recorder.playBookmark());
        } else {
          output.init(source);
          int mindex = recorder.reserveMatchIndex(level, element, source.getBookmark(), parentNot, elementNot);
          int bindex = recorder.reserveBookmarkIndex();
          match = element.parse(level, source, not|parentNot|elementNot, caseSensitive, false);
          match = match != elementNot;
          recorder.set(level, source, element, bookmark, source.getBookmark(), source.getBestIndex(), mindex, bindex, match, parentNot, elementNot);
          output.set(source, bookmark);
        }
      } else {
        if (receive) {
          match = recorder.playMatch(element, source.getBookmark());
        } else {
          int index = recorder.reserveMatchIndex(level, element, source.getBookmark(), parentNot, elementNot);
          match = element.parse(level, source, not|parentNot|elementNot, caseSensitive, false);
          match = (match != elementNot);
          recorder.set(level, source, element, bookmark, source.getBookmark(), source.getBestIndex(), index, match, parentNot, elementNot);
        }
      }


      if (match) {
        if (receive && !elementNot && !not) {
          source.setBookmark(bookmark);
          output.init(source);
          element.parse(level, source, not|parentNot|elementNot, caseSensitive, receive);
          output.set(source, bookmark);
        }
                output = list.getOutput();
                if (output != null && receive) {
                    output.init(source);
                    output.set(source, bookmark);
                }
                
                return !not;
      }
    }
    if (not) {
      source.setBookmark(bookmark);
      source.skipCharacter();


      Output output = list.getOutput();
      if (output != null && receive) {
        output.init(source);
        output.set(source, bookmark);
      }
    } else {
      recorder.setIndex(recIndex);
    }

View Full Code Here

    int startIndex = recorder.getIndex();
    
    for (Iterator i = list.iterator(); i.hasNext();) {
      ElementOutput elementOutput = (ElementOutput)i.next();
      Element element = elementOutput.getElement();
      Output output = elementOutput.getOutput();
      boolean elementNot = elementOutput.isNot();


      if (this.caseSensitive != null) {
        caseSensitive = this.caseSensitive.booleanValue();
      }
      if (elementOutput.isCaseSensitive() != null) {
        caseSensitive = elementOutput.isCaseSensitive().booleanValue();
      }


      int bookmark = source.getBookmark();


      boolean match = false;
      if (output.receive()) {
        if (receive) {
          match = recorder.playMatch(element, source.getBookmark());
          source.setBookmark(recorder.playBookmark());
        } else {
          output.init(source);
          int mindex = recorder.reserveMatchIndex(level, element, source.getBookmark(), parentNot, elementNot);
          int bindex = recorder.reserveBookmarkIndex();
          match = element.parse(level, source, not|parentNot|elementNot, caseSensitive, false);
          match = match != elementNot;
          recorder.set(level, source, element, bookmark, source.getBookmark(), source.getBestIndex(), mindex, bindex, match, parentNot, elementNot);
          output.set(source, bookmark);
        }
      } else {
        if (receive) {
          match = recorder.playMatch(element, source.getBookmark());
        } else {
          int mindex = recorder.reserveMatchIndex(level, element, source.getBookmark(), parentNot, elementNot);
          match = element.parse(level, source, not|parentNot|elementNot, caseSensitive, false);
          match = match != elementNot;
          recorder.set(level, source, element, bookmark, source.getBookmark(), source.getBestIndex(), mindex, match, parentNot, elementNot);
        }
      }
      
      if (match == not) {
        if (!receive) {
          recorder.setIndex(startIndex);
        }
        return false;
      }
      
      if (not) {
        break;
      }


      if (elementNot) {
        source.setBookmark(bookmark);
        source.skipCharacter();
        if (source.getBookmark() > bestMatchIdx) {
          source.setBestIndex(source.getBookmark());
        }
      }


      if (receive) {
        if (elementNot) {
          output.init(source);
          output.set(source, bookmark);
        } else {
          source.setBookmark(bookmark);
          output.init(source);
          element.parse(level, source, not|parentNot|elementNot, caseSensitive, receive);
          output.set(source, bookmark);
        }
      }
    }


    if (not) {

View Full Code Here

0 1 2 3 4 5 6 7

TOP

Related Classes of com.datasalt.pangool.tuplemr.TupleMRBuilder$Output

com.datasalt.pangool.examples.avro.AvroCustomSerializationJob

com.datasalt.pangool.examples.avro.AvroTopicalWordCount

com.datasalt.pangool.examples.avro.AvroTweetsJoin

com.datasalt.pangool.examples.gameoflife.GameOfLifeJob

com.datasalt.pangool.examples.movingaverage.MovingAverage

com.datasalt.pangool.examples.naivebayes.NaiveBayesGenerate

com.datasalt.pangool.examples.secondarysort.SecondarySort

com.datasalt.pangool.examples.simplesecondarysort.SimpleSecondarySort

com.datasalt.pangool.examples.solr.MultiShakespeareIndexer

com.datasalt.pangool.examples.topicalwordcount.TopicalWordCount

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.