Examples of GoogleCloudStorageFileOutputWriter

com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutputWriter
An {@link OutputWriter} that writes bytes to a GCS file that it creates.Produces a single file output (usually on a per-shard basis). This format does not insert any separator characters, so it by default cannot be read back with the CloudStorageLineInputReader.

Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutputWriter


  }


  public void writeData(GcsFilename filename, ByteBufferGenerator gen) throws IOException {
    LevelDbOutputWriter writer = new GoogleCloudStorageLevelDbOutputWriter(
        new GoogleCloudStorageFileOutputWriter(filename, "TestData"));
    writer.beginShard();
    writer.beginSlice();
    while (gen.hasNext()) {
      writer.write(gen.next());
    }

View Full Code Here

Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutputWriter

      Marshaller<ByteBuffer> identity = Marshallers.getByteBufferMarshaller();
      // Uses LevelDbOutputWriter wrapping GoogleCloudStorageFileOutputWriter rather than
      // GoogleCloudStorageLevelDbOutputWriter because the padding at the end of the slice is
      // unneeded as the file is being finalized.
      return new MarshallingOutputWriter<>(
          new LevelDbOutputWriter(new GoogleCloudStorageFileOutputWriter(
              new GcsFilename(bucket, fileName), MapReduceConstants.REDUCE_INPUT_MIME_TYPE)),
          Marshallers.getKeyValuesMarshaller(identity, identity));
    }

View Full Code Here

Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutputWriter

    @Override
    public OutputWriter<KeyValue<K, V>> createWriter(int sortShard) {
      GcsFilename file = new GcsFilename(bucket, getFileName(sortShard));
      MarshallingOutputWriter<KeyValue<K, V>> output =
          new MarshallingOutputWriter<>(new GoogleCloudStorageLevelDbOutputWriter(
              new GoogleCloudStorageFileOutputWriter(file, MAP_OUTPUT_MIME_TYPE)),
              new KeyValueMarshaller<>(keyMarshaller, valueMarshaller));
      return output;
    }

View Full Code Here

Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutputWriter

        int sliceNumber) {
      Marshaller<ByteBuffer> identity = Marshallers.getByteBufferMarshaller();
      String fileName = String.format(fileNamePattern, sliceNumber);
      fileNames.add(fileName);
      return new MarshallingOutputWriter<>(
          new GoogleCloudStorageLevelDbOutputWriter(new GoogleCloudStorageFileOutputWriter(
              new GcsFilename(bucket, fileName), MapReduceConstants.REDUCE_INPUT_MIME_TYPE)),
          Marshallers.getKeyValuesMarshaller(identity, identity));
    }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.