Examples of PureJavaCrc32

com.alimama.mdrill.editlog.util.PureJavaCrc32
org.apache.cassandra.utils.PureJavaCrc32
A pure-java implementation of the CRC32 checksum that uses the same polynomial as the built-in native CRC32. This is to avoid the JNI overhead for certain uses of Checksumming where many small pieces of data are checksummed in succession. The current version is ~10x to 1.8x as fast as Sun's native java.util.zip.CRC32 in Java 1.6 @see java.util.zip.CRC32 This class is copied from hadoop-commons project. (The initial patch added PureJavaCrc32 was HADOOP-6148)
org.apache.hadoop.util.PureJavaCrc32
A pure-java implementation of the CRC32 checksum that uses the same polynomial as the built-in native CRC32. This is to avoid the JNI overhead for certain uses of Checksumming where many small pieces of data are checksummed in succession. The current version is ~10x to 1.8x as fast as Sun's native java.util.zip.CRC32 in Java 1.6 @see java.util.zip.CRC32
org.apache.hadoop_voltpatches.util.PureJavaCrc32
A pure-java implementation of the CRC32 checksum that uses the same polynomial as the built-in native CRC32. This is to avoid the JNI overhead for certain uses of Checksumming where many small pieces of data are checksummed in succession. The current version is ~10x to 1.8x as fast as Sun's native java.util.zip.CRC32 in Java 1.6 @see java.util.zip.CRC32

Examples of org.apache.hadoop.util.PureJavaCrc32

     */
    @SuppressWarnings("deprecation")
    public Reader(DataInputStream in, int logVersion) {
      this.logVersion = logVersion;
      if (LayoutVersion.supports(Feature.EDITS_CHESKUM, logVersion)) {
        this.checksum = new PureJavaCrc32();
      } else {
        this.checksum = null;
      }


      if (this.checksum != null) {

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

    return value;
  }


  private DFSOutputStream(DFSClient dfsClient, String src, long blockSize, Progressable progress,
      int bytesPerChecksum, short replication) throws IOException {
    super(new PureJavaCrc32(), bytesPerChecksum, 4);
    this.dfsClient = dfsClient;
    this.conf = dfsClient.conf;
    this.src = src;
    this.blockSize = blockSize;
    this.blockReplication = replication;

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

        byte[] version = new byte[CHECKSUM_VERSION.length];
        sums.readFully(version);
        if (!Arrays.equals(version, CHECKSUM_VERSION))
          throw new IOException("Not a checksum file: "+sumFile);
        this.bytesPerSum = sums.readInt();
        set(fs.verifyChecksum, new PureJavaCrc32(), bytesPerSum, 4);
      } catch (FileNotFoundException e) {         // quietly ignore
        set(fs.verifyChecksum, null, 1, 0);
      } catch (IOException e) {                   // loudly ignore
        LOG.warn("Problem opening checksum file: "+ file + 
                 ".  Ignoring exception: " +

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

                          int bufferSize,
                          short replication,
                          long blockSize,
                          Progressable progress)
      throws IOException {
      super(new PureJavaCrc32(), fs.getBytesPerSum(), 4);
      int bytesPerSum = fs.getBytesPerSum();
      this.datas = fs.getRawFileSystem().create(file, overwrite, bufferSize, 
                                         replication, blockSize, progress);
      int sumBufferSize = fs.getSumBufferSize(bytesPerSum, bufferSize);
      this.sums = fs.getRawFileSystem().create(fs.getChecksumFile(file), true,

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

        sums.readFully(version);
        if (!Arrays.equals(version, CHECKSUM_VERSION)) {
          throw new IOException("Not a checksum file: "+sumFile);
        }
        this.bytesPerSum = sums.readInt();
        set(fs.verifyChecksum, new PureJavaCrc32(), bytesPerSum, 4);
      } catch (FileNotFoundException e) {         // quietly ignore
        set(fs.verifyChecksum, null, 1, 0);
      } catch (IOException e) {                   // loudly ignore
        LOG.warn("Problem opening checksum file: "+ file + 
                 ".  Ignoring exception: " +

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

      final EnumSet<CreateFlag> createFlag,
      final FsPermission absolutePermission, final int bufferSize,
      final short replication, final long blockSize, 
      final Progressable progress, final int bytesPerChecksum,
      final boolean createParent) throws IOException {
      super(new PureJavaCrc32(), fs.getBytesPerSum(), 4);


      this.datas = fs.getRawFs().createInternal(file, createFlag,
          absolutePermission, bufferSize, replication, blockSize, progress,
           bytesPerChecksum,  createParent);

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

        numPartitions * MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH);
    entries = buf.asLongBuffer();
  }


  public SpillRecord(Path indexFileName, JobConf job) throws IOException {
    this(indexFileName, job, new PureJavaCrc32());
  }

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

  /**
   * Write this spill record to the location provided.
   */
  public void writeToFile(Path loc, JobConf job)
      throws IOException {
    writeToFile(loc, job, new PureJavaCrc32());
  }

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

    } finally {
      IOUtils.closeStream(instr);
    }


    // compute crc of partial chunk from data read in the block file.
    partialCrc = new PureJavaCrc32();
    partialCrc.update(buf, 0, sizePartialChunk);
    LOG.info("Read in partial CRC chunk from disk for block " + block);


    // paranoia! verify that the pre-computed crc matches what we
    // recalculated just now

View Full Code Here

Examples of org.apache.hadoop.util.PureJavaCrc32

    private final DataOutputBuffer buf;
    private final Checksum checksum;


    public Writer(DataOutputBuffer out) {
      this.buf = out;
      this.checksum = new PureJavaCrc32();
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.