Examples of org.apache.hadoop.mapred.PhasedFileSystem$FileInfo

org.apache.hadoop.mapred.PhasedFileSystem
This class acts as a proxy to the actual file system being used. It writes files to a temporary location and on commit, moves to final location. On abort or a failure in commit the temporary file is deleted PhasedFileSystem works in context of a task. A different instance of PhasedFileSystem should be used for every task. Temporary files are written in ("mapred.system.dir")// If one tasks opens a large number of files in succession then its better to commit(Path) individual files when done. Otherwise commit() can be used to commit all open files at once. @deprecated {@link PhasedFileSystem} is no longer usedduring speculative execution of tasks.

        // client has renamed outputPath and saved the argv's original output path as:
        if (useSingleSideOutputURI_) {
          finalOutputURI = new URI(sideOutputURI_);
          sideEffectPathFinal_ = null; // in-place, no renaming to final
        } else {
          sideFs_ = new PhasedFileSystem(sideFs_, job);
          String sideOutputPath = job_.get("stream.sideoutput.dir"); // was: job_.getOutputPath() 
          String fileName = getSideEffectFileName(); // see HADOOP-444 for rationale
          sideEffectPathFinal_ = new Path(sideOutputPath, fileName);
          finalOutputURI = new URI(sideEffectPathFinal_.toString()); // implicit dfs: 
        }

View Full Code Here

        // client has renamed outputPath and saved the argv's original output path as:
        if (useSingleSideOutputURI_) {
          finalOutputURI = new URI(sideOutputURI_);
          sideEffectPathFinal_ = null; // in-place, no renaming to final
        } else {
          sideFs_ = new PhasedFileSystem(sideFs_, job);
          String sideOutputPath = job_.get("stream.sideoutput.dir"); // was: job_.getOutputPath() 
          String fileName = getSideEffectFileName(); // see HADOOP-444 for rationale
          sideEffectPathFinal_ = new Path(sideOutputPath, fileName);
          finalOutputURI = new URI(sideEffectPathFinal_.toString()); // implicit dfs: 
        }

View Full Code Here

        // client has renamed outputPath and saved the argv's original output path as:
        if (useSingleSideOutputURI_) {
          finalOutputURI = new URI(sideOutputURI_);
          sideEffectPathFinal_ = null; // in-place, no renaming to final
        } else {
          sideFs_ = new PhasedFileSystem(sideFs_, job);
          String sideOutputPath = job_.get("stream.sideoutput.dir"); // was: job_.getOutputPath() 
          String fileName = getSideEffectFileName(); // see HADOOP-444 for rationale
          sideEffectPathFinal_ = new Path(sideOutputPath, fileName);
          finalOutputURI = new URI(sideEffectPathFinal_.toString()); // implicit dfs: 
        }

View Full Code Here

TOP

Related Classes of org.apache.hadoop.mapred.PhasedFileSystem$FileInfo

org.apache.hadoop.fs.FSDataOutputStream

org.apache.hadoop.fs.FSOutputStream

org.apache.hadoop.fs.Path

org.apache.hadoop.streaming.PipeMapRed

org.apache.lucene.util.BytesRef

org.apache.lucene.util.Version

org.elasticsearch.common.unit.ByteSizeValue

org.elasticsearch.index.store.StoreFileMetaData

ro.redeul.google.go.lang.psi.GoFile

org.elasticsearch.ElasticsearchParseException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.