Examples of com.clearspring.analytics.stream.cardinality.HyperLogLog

com.clearspring.analytics.stream.cardinality.HyperLogLog

inria.fr/flajolet/Publications/FlFuGaMe07.pdf

HLL is an improved version of LogLog that is capable of estimating the cardinality of a set with accuracy = 1.04/sqrt(m) where m = 2^b. So we can control accuracy vs space usage by increasing or decreasing b.

The main benefit of using HLL over LL is that it only requires 64% of the space that LL does to get the same accuracy.

This implementation implements a single counter. If a large (millions) number of counters are required you may want to refer to:

http://dsiutils.dsi.unimi.it/

It has a more complex implmentation of HLL that supports multiple counters in a single object, drastically reducing the java overhead from creating a large number of objects.

This implementation leveraged a javascript implementation that Yammer has been working on:

https://github.com/yammer/probablyjs

    }


    @Override
    public void prepare(FlowProcess flowProcess, OperationCall operationCall) {
      JobConf conf = (JobConf) flowProcess.getConfigCopy();
      approxCounter = new HyperLogLog(BloomProps.getHllErr(conf));
      sampleRate = BloomProps.getKeySampleRate(conf);
      tupleSerializationUtil = new TupleSerializationUtil((JobConf) flowProcess.getConfigCopy());
    }

    List<HyperLogLog> countParts = new LinkedList<HyperLogLog>();


    long totalSum = 0;
    while (in.hasNext()) {
      TupleEntry tuple = in.next();
      HyperLogLog card = HyperLogLog.Builder.build(Bytes.getBytes((BytesWritable) tuple.getObject("bytes")));
      countParts.add(card);
      totalSum += card.cardinality();
    }


    HyperLogLog merged = (HyperLogLog) new HyperLogLog(BloomProps.getHllErr(conf)).merge(countParts.toArray(new ICardinality[countParts.size()]));
    long cardinality = merged.cardinality();


    //  HLL estimation doesn't work over 2^32, and the cardinality code just returns 0.
    //  Honestly if you get this high, your bloom filter is probably saturated anyway, so just return that max.
    if (cardinality == 0 && totalSum != 0) {
      LOG.info("HyperLogLog likely reached its max estimation of 2^32! Returning that max, but true count likely higher.");

Examples of com.clearspring.analytics.stream.cardinality.HyperLogLog

Related Classes of com.clearspring.analytics.stream.cardinality.HyperLogLog