Examples of TezRawKeyValueIterator

org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator
TezRawKeyValueIterator is an iterator used to iterate over the raw keys and values during sort/merge of intermediate data.

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

              ++spindex;
            }
            // Note: we would like to avoid the combiner if we've fewer
            // than some threshold of records for a partition
            if (spstart != spindex) {
              TezRawKeyValueIterator kvIter =
                new MRResultIterator(spstart, spindex);
              if (LOG.isDebugEnabled()) {
                LOG.debug("Running combine processor");
              }
              runCombineProcessor(kvIter, writer);

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

            this.conf.getInt(TezJobConfig.TEZ_RUNTIME_IO_SORT_FACTOR,
                TezJobConfig.DEFAULT_TEZ_RUNTIME_IO_SORT_FACTOR);
        // sort the segments only if there are intermediate merges
        boolean sortSegments = segmentList.size() > mergeFactor;
        //merge
        TezRawKeyValueIterator kvIter = TezMerger.merge(conf, rfs,
                       keyClass, valClass, codec,
                       segmentList, mergeFactor,
                       new Path(taskIdentifier),
                       (RawComparator)ConfigUtils.getIntermediateOutputKeyComparator(conf),
                       nullProgressable, sortSegments, true,

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

   * @throws InterruptedException
   */
  public TezRawKeyValueIterator waitForInput() throws IOException, InterruptedException {
    Preconditions.checkState(runShuffleFuture != null,
        "waitForInput can only be called after run");
    TezRawKeyValueIterator kvIter;
    try {
      kvIter = runShuffleFuture.get();
    } catch (ExecutionException e) {
      Throwable cause = e.getCause();
      if (cause instanceof IOException) {

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

      // stop the scheduler
      scheduler.close();




      // Finish the on-going merges...
      TezRawKeyValueIterator kvIter = null;
      try {
        kvIter = merger.close();
      } catch (Throwable e) {
        inputContext.fatalError(e, "Error while doing final merge ");
        throw new ShuffleError("Error while doing final merge " , e);

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

  @Private
  public TezRawKeyValueIterator getIterator() throws IOException, InterruptedException {
    // wait for input so that iterator is available
    synchronized(this) {
    if (this.numInputs == 0) {
      return new TezRawKeyValueIterator() {
        @Override
        public DataInputBuffer getKey() throws IOException {
          throw new RuntimeException("No data available in Input");
        }

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

        return;
      }
      localShuffleCopy = shuffle;
    }
    
    TezRawKeyValueIterator localRawIter = localShuffleCopy.waitForInput();
    synchronized(this) {
      rawIter = localRawIter;
      createValuesIterator();
    }
  }

View Full Code Here

Examples of org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator

   * @return a KVReader over the sorted input.
   */
  @Override
  public KeyValuesReader getReader() throws IOException {
    // Cannot synchronize entire method since this is called form user code and can block.
    TezRawKeyValueIterator rawIterLocal;
    synchronized (this) {
      rawIterLocal = rawIter;
      if (this.numInputs == 0) {
        return new KeyValuesReader() {
          @Override

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.