Examples of Lfs

cascading.tap.Lfs
cascading.tap.hadoop.Lfs
Class Lfs is a {@link cascading.tap.Tap} class that provides access to the Local File System via Hadoop.
Note that using a Lfs {@link cascading.tap.Tap} instance in a {@link cascading.flow.Flow} will force a portion of not the whole Flow to be executedin "local" mode forcing the Flow to execute in the current JVM. Mixing with {@link cascading.tap.hadoop.Dfs} and other Taptypes is possible, providing a means to implement complex file/data management functions.
Use {@link cascading.tap.hadoop.Hfs} if you need a Tap instance that inherits the default {@link FileSystem} used by Hadoop.

Examples of cascading.tap.hadoop.Lfs

    }


  @Test
  public void testCopyConfig() throws Exception
    {
    Tap source = new Lfs( new TextLine(), "input/path" );
    Tap sink = new Hfs( new TextLine(), "output/path", true );


    Pipe pipe = new Pipe( "test" );


    Configuration conf = ( (BaseHadoopPlatform) getPlatform() ).getConfiguration();

View Full Code Here

Examples of cascading.tap.hadoop.Lfs

        defaults.set("parquet.benchmark.bytes.total", "false");
        defaults.set("parquet.benchmark.time.read", "false");


        ((LocalScheme<SourceCtx, SinkCtx>) this.getScheme()).setDefaults(defaults);


        lfs = new Lfs(scheme, path);
        ((LocalScheme<SourceCtx, SinkCtx>) this.getScheme()).setLfs(lfs);
    }

View Full Code Here

Examples of cascading.tap.hadoop.Lfs

      new Fields("in"));


    Pipe pipe = new Pipe("batch-pipe");
    pipe = new Each(pipe, new AnnotateWithSizeOfCurrentBatch(4));


    Tap<JobConf, RecordReader, OutputCollector> dst = new Lfs(new SequenceFile(OUT_FIELD), getTestRoot()+"/out");


    Flow f = CascadingUtil.get().getFlowConnector().connect(src, dst, pipe);
    f.complete();


    TupleEntryIterator tupleEntryIterator = dst.openForRead(CascadingUtil.get().getFlowProcess());
    List<BytesWritable> outStrings = new ArrayList<BytesWritable>(6);
    while (tupleEntryIterator.hasNext()) {
      TupleEntry next = tupleEntryIterator.next();
      BytesWritable bytes = (BytesWritable) next.getObject(OUT_FIELD);
      outStrings.add(bytes);

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.