Examples of ByteBloomFilter

com.taobao.top.analysis.util.bloom.ByteBloomFilter
@author fangweng @email fangweng@taobao.com @date 2011-3-16Implements a Bloom filter, as defined by Bloom in 1970. The Bloom filter is a data structure that was introduced in 1970 and that has been adopted by the networking research community in the past decade thanks to the bandwidth efficiencies that it offers for the transmission of set membership information between networked hosts. A sender encodes the information into a bit vector, the Bloom filter, that is more compact than a conventional representation. Computation and space costs for construction are linear in the number of elements. The receiver uses the filter to test whether various elements are members of the set. Though the filter will occasionally return a false positive, it will never return a false negative. When creating the filter, the sender can choose its desired point in a trade-off between the false positive rate and the size.
org.apache.hadoop.hbase.util.ByteBloomFilter
ne-lab.org">European Commission One-Lab Project 034819. Bloom filters are very sensitive to the number of elements inserted into them. For HBase, the number of entries depends on the size of the data stored in the column. Currently the default region size is 256MB, so entry count ~= 256MB / (average value size for column). Despite this rule of thumb, there is no efficient way to calculate the entry count after compactions. Therefore, it is often easier to use a dynamic bloom filter that will add extra space instead of allowing the error rate to grow. ( http://www.eecs.harvard.edu/~michaelm/NEWWORK/postscripts/BloomFilterSurvey .pdf ) m denotes the number of bits in the Bloom filter (bitSize) n denotes the number of elements inserted into the Bloom filter (maxKeys) k represents the number of hash functions used (nbHash) e represents the desired false positive rate for the bloom (err) If we fix the error rate (e) and know the number of entries, then the optimal bloom size m = -(n * ln(err) / (ln(2)^2) ~= n * ln(err) / ln(0.6185) The probability of false positives is minimized when k = m/n ln(2). @see BloomFilter The general behavior of a filter @see Space/Time Trade-Offs in Hash Coding with Allowable Errors

Examples of com.taobao.top.analysis.util.bloom.ByteBloomFilter

  }
  
  DistinctCountEntryValue createDCEntryValue(ReportEntry entry)
  {
    DistinctCountEntryValue distinctEntryValue = new DistinctCountEntryValue();
    ByteBloomFilter bloomFilter;
    
    //240k
    int maxKeys = 100000;
    float errorRate = 0.0001F;
    
    if (entry.getAdditions().get(AnalysisConstants.ANALYSIS_BLOOM_MAXKEYS) != null)
      maxKeys = (Integer)entry.getAdditions().get(AnalysisConstants.ANALYSIS_BLOOM_MAXKEYS);
    
    if (entry.getAdditions().get(AnalysisConstants.ANALYSIS_BLOOM_ERRORRATE) != null)
      errorRate = (Float)entry.getAdditions().get(AnalysisConstants.ANALYSIS_BLOOM_ERRORRATE);
      
    bloomFilter = new ByteBloomFilter(maxKeys,errorRate,1);
    
    distinctEntryValue.setBloomFilter(bloomFilter);
    
    return distinctEntryValue;
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

        ByteBuffer b = getMetaBlock(BLOOM_FILTER_META_KEY, false);
        if (b != null) {
          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }
          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType==BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + name);
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter


      if (bloomType != BloomType.NONE && conf != null) {
        float err = conf.getFloat("io.hfile.bloom.error.rate", (float)0.01);
        int maxFold = conf.getInt("io.hfile.bloom.max.fold", 7);


        this.bloomFilter = new ByteBloomFilter(maxKeys, err,
            Hash.getHashType(conf), maxFold);
        this.bloomFilter.allocBloom();
        this.bloomType = bloomType;
      } else {
        this.bloomFilter = null;

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);
        int tooBig = conf.getInt(IO_STOREFILE_BLOOM_MAX_KEYS, 128*1000*1000);
        
        if (maxKeys < tooBig) { 
          try {
            bloom = new ByteBloomFilter(maxKeys, err,
                Hash.getHashType(conf), maxFold);
            bloom.allocBloom();
            bt = bloomType;
          } catch (IllegalArgumentException iae) {
            LOG.warn(String.format(

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);
        int tooBig = conf.getInt(IO_STOREFILE_BLOOM_MAX_KEYS, 128*1000*1000);
        
        if (maxKeys < tooBig) { 
          try {
            bloom = new ByteBloomFilter(maxKeys, err,
                Hash.getHashType(conf), maxFold);
            bloom.allocBloom();
            bt = bloomType;
          } catch (IllegalArgumentException iae) {
            LOG.warn(String.format(

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

        if (bloomType == BloomType.ROWCOL) {
          err /= 2;
        }
        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);


        this.bloomFilter = new ByteBloomFilter(maxKeys, err,
            Hash.getHashType(conf), maxFold);
        this.bloomFilter.allocBloom();
        this.bloomType = bloomType;
      } else {
        this.bloomFilter = null;

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

Examples of org.apache.hadoop.hbase.util.ByteBloomFilter

          }


          //Printing bloom information
          ByteBuffer b = reader.getMetaBlock("BLOOM_FILTER_META", false);
          if (b!= null) {
            BloomFilter bloomFilter = new ByteBloomFilter(b);
            System.out.println("BloomSize: " + bloomFilter.getByteSize());
            System.out.println("No of Keys in bloom: " +
                bloomFilter.getKeyCount());
            System.out.println("Max Keys for bloom: " +
                bloomFilter.getMaxKeys());
          } else {
            System.out.println("Could not get bloom data from meta block");
          }
        }
        if (printBlocks) {

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.