Examples of ThrottledInputStream

net.sf.katta.util.ThrottledInputStream
es an throttled input stream which reads 1024 bytes/sec from the underlying input stream at the most
ThrottledInputStream throttledInputStream = new ThrottledInputStream(otherIputStream, new ThrottleSemaphore(1024));

Usage over multiple {@link InputStream}s:
//throttle the read of multiple input streams at the rate of 1024 bytes/sec
ThrottleSemaphore semaphore = new ThrottleSemaphore(1024); ThrottledInputStream throttledInputStream1 = new ThrottledInputStream(otherIputStream1, semaphore); ThrottledInputStream throttledInputStream2 = new ThrottledInputStream(otherIputStream2, semaphore); ...
org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream
The ThrottleInputStream provides bandwidth throttling on a specified InputStream. It is implemented as a wrapper on top of another InputStream instance. The throttling works by examining the number of bytes read from the underlying InputStream from the beginning, and sleep()ing for a time interval if the byte-transfer is found exceed the specified tolerable maximum. (Thus, while the read-rate might exceed the maximum for a given short interval, the average tends towards the specified maximum, overall.)
org.apache.hadoop.tools.distcp2.util.ThrottledInputStream
The ThrottleInputStream provides bandwidth throttling on a specified InputStream. It is implemented as a wrapper on top of another InputStream instance. The throttling works by examining the number of bytes read from the underlying InputStream from the beginning, and sleep()ing for a time interval if the byte-transfer is found exceed the specified tolerable maximum. (Thus, while the read-rate might exceed the maximum for a given short interval, the average tends towards the specified maximum, overall.)
org.apache.hadoop.tools.util.ThrottledInputStream
The ThrottleInputStream provides bandwidth throttling on a specified InputStream. It is implemented as a wrapper on top of another InputStream instance. The throttling works by examining the number of bytes read from the underlying InputStream from the beginning, and sleep()ing for a time interval if the byte-transfer is found exceed the specified tolerable maximum. (Thus, while the read-rate might exceed the maximum for a given short interval, the average tends towards the specified maximum, overall.)
xnap.io.ThrottledInputStream

Examples of net.sf.katta.util.ThrottledInputStream

      return _fileSystemDelegate.mkdirs(arg0, arg1);
    }


    @Override
    public FSDataInputStream open(Path path, int bufferSize) throws IOException {
      ThrottledInputStream throttledInputStream = new ThrottledInputStream(_fileSystemDelegate.open(path, bufferSize),
              _throttleSemaphore);
      return new FSDataInputStream(throttledInputStream);
    }

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputPath);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }
      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());


        // Ensure that the output folder is there and copy the file

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputInfo);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }


      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputInfo);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }


      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputPath);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }
      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());


        // Ensure that the output folder is there and copy the file

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputPath);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }
      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());


        // Ensure that the output folder is there and copy the file

View Full Code Here

Examples of org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream

      }


      InputStream in = openSourceFile(context, inputInfo);
      int bandwidthMB = context.getConfiguration().getInt(CONF_BANDWIDTH_MB, 100);
      if (Integer.MAX_VALUE != bandwidthMB) {
        in = new ThrottledInputStream(new BufferedInputStream(in), bandwidthMB * 1024 * 1024);
      }


      try {
        context.getCounter(Counter.BYTES_EXPECTED).increment(inputStat.getLen());

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.util.ThrottledInputStream

  private long copyBytes(FileStatus sourceFileStatus, OutputStream outStream,
                         int bufferSize, Mapper.Context context)
      throws IOException {
    Path source = sourceFileStatus.getPath();
    byte buf[] = new byte[bufferSize];
    ThrottledInputStream inStream = null;
    long totalBytesRead = 0;


    try {
      inStream = getInputStream(source, context.getConfiguration());
      int bytesRead = readBytes(inStream, buf);
      while (bytesRead >= 0) {
        totalBytesRead += bytesRead;
        outStream.write(buf, 0, bytesRead);
        updateContextStatus(totalBytesRead, context, sourceFileStatus);
        bytesRead = inStream.read(buf);
      }
    } finally {
      IOUtils.cleanup(LOG, outStream, inStream);
    }

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.util.ThrottledInputStream

          throws IOException {
    try {
      FileSystem fs = path.getFileSystem(conf);
      long bandwidthMB = conf.getInt(DistCpConstants.CONF_LABEL_BANDWIDTH_MB,
              DistCpConstants.DEFAULT_BANDWIDTH_MB);
      return new ThrottledInputStream(new BufferedInputStream(fs.open(path)),
              bandwidthMB * 1024 * 1024);
    }
    catch (IOException e) {
      throw new CopyReadException(e);
    }

View Full Code Here

Examples of org.apache.hadoop.tools.util.ThrottledInputStream

  long copyBytes(FileStatus sourceFileStatus, long sourceOffset,
      OutputStream outStream, int bufferSize, Mapper.Context context)
      throws IOException {
    Path source = sourceFileStatus.getPath();
    byte buf[] = new byte[bufferSize];
    ThrottledInputStream inStream = null;
    long totalBytesRead = 0;


    try {
      inStream = getInputStream(source, context.getConfiguration());
      int bytesRead = readBytes(inStream, buf, sourceOffset);

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.