Examples of org.apache.lucene.util.OpenBitSet

org.apache.lucene.util.OpenBitSet
An "open" BitSet implementation that allows direct access to the array of words storing the bits.
Unlike java.util.bitset, the fact that bits are packed into an array of longs is part of the interface. This allows efficient implementation of other algorithms by someone other than the author. It also allows one to efficiently implement alternate serialization or interchange formats.
OpenBitSet is faster than java.util.BitSet in most operations and *much* faster at calculating cardinality of sets and results of set operations. It can also handle sets of larger cardinality (up to 64 * 2**32-1)
The goals of OpenBitSet are the fastest implementation possible, and maximum code reuse. Extra safety and encapsulation may always be built on top, but if that's built in, the cost can never be removed (and hence people re-implement their own version in order to get better performance). If you want a "safe", totally encapsulated (and slower and limited) BitSet class, use java.util.BitSet.

Performance Results
Test system: Pentium 4, Sun Java 1.5_06 -server -Xbatch -Xmx64M
BitSet size = 1,000,000
Results are java.util.BitSet time divided by OpenBitSet time.

cardinality intersect_count union nextSetBit get iterator

50% full 3.36 3.96 1.44 1.46 1.99 1.58

1% full 3.31 3.90 1.04 0.99

Test system: AMD Opteron, 64 bit linux, Sun Java 1.5_06 -server -Xbatch -Xmx64M
BitSet size = 1,000,000
Results are java.util.BitSet time divided by OpenBitSet time.

cardinality intersect_count union nextSetBit get iterator

50% full 2.50 3.50 1.00 1.03 1.12 1.25

1% full 2.51 3.49 1.00 1.02

@version $Id$

	cardinality	intersect_count	union	nextSetBit	get	iterator
50% full	3.36	3.96	1.44	1.46	1.99	1.58
1% full	3.31	3.90		1.04		0.99

	cardinality	intersect_count	union	nextSetBit	get	iterator
50% full	2.50	3.50	1.00	1.03	1.12	1.25
1% full	2.51	3.49		1.00		1.02

    }
  }


  @Override
   public DocSet andNot(DocSet other) {
    OpenBitSet newbits = (OpenBitSet)(bits.clone());
     if (other instanceof BitDocSet) {
       newbits.andNot(((BitDocSet)other).bits);
     } else {
       DocIterator iter = other.iterator();
       while (iter.hasNext()) newbits.clear(iter.nextDoc());
     }
     return new BitDocSet(newbits);
  }

View Full Code Here

     return new BitDocSet(newbits);
  }


  @Override
   public DocSet union(DocSet other) {
     OpenBitSet newbits = (OpenBitSet)(bits.clone());
     if (other instanceof BitDocSet) {
       newbits.union(((BitDocSet)other).bits);
     } else {
       DocIterator iter = other.iterator();
       while (iter.hasNext()) newbits.set(iter.nextDoc());
     }
     return new BitDocSet(newbits);
  }

View Full Code Here

      boolean doNegative = baseSize > maxDoc >> 1 && termInstances > 0
              && startTerm==0 && endTerm==numTermsInField
              && docs instanceof BitDocSet;


      if (doNegative) {
        OpenBitSet bs = (OpenBitSet)((BitDocSet)docs).getBits().clone();
        bs.flip(0, maxDoc);
        // TODO: when iterator across negative elements is available, use that
        // instead of creating a new bitset and inverting.
        docs = new BitDocSet(bs, maxDoc - baseSize);
        // simply negating will mean that we have deleted docs in the set.
        // that should be OK, as their entries in our table should be empty.

View Full Code Here

      doNegative = baseSize > maxDoc >> 1 && termInstances > 0
              && docs instanceof BitDocSet;
    }


    if (doNegative) {
      OpenBitSet bs = (OpenBitSet) ((BitDocSet) docs).getBits().clone();
      bs.flip(0, maxDoc);
      // TODO: when iterator across negative elements is available, use that
      // instead of creating a new bitset and inverting.
      docs = new BitDocSet(bs, maxDoc - baseSize);
      // simply negating will mean that we have deleted docs in the set.
      // that should be OK, as their entries in our table should be empty.

View Full Code Here


    public void addDocumentNormalizations(LucandraTermInfo[] allDocs, String field, ReaderCache cache)
    {


        byte[] norms = cache.fieldNorms.get(field);
        OpenBitSet docHits = cache.docHits;


        for (LucandraTermInfo docInfo : allDocs)
        {


            int idx = docInfo.docId;


            if (idx > numDocs)
                throw new IllegalStateException("numDocs reached");


            Byte norm = docInfo.norm;


            if (norm == null)
                norm = defaultNorm;


            // Check for cached reads
            if (norms != null && norms.length > idx && norms[idx] == norm)
                continue;


            docHits.fastSet(idx);


            if (norms == null)
                norms = new byte[numDocs];


            norms[idx] = norm;

View Full Code Here

    if (pos < scratch.length) {
      scratch[pos]=doc;
    } else {
      // this conditional could be removed if BitSet was preallocated, but that
      // would take up more memory, and add more GC time...
      if (bits==null) bits = new OpenBitSet(maxDoc);
      bits.fastSet(doc);
    }


    pos++;
  }

View Full Code Here

    if (pos < scratch.length) {
      scratch[pos]=doc;
    } else {
      // this conditional could be removed if BitSet was preallocated, but that
      // would take up more memory, and add more GC time...
      if (bits==null) bits = new OpenBitSet(maxDoc);
      bits.fastSet(doc);
    }


    pos++;
  }

View Full Code Here


          // calculate the maximum value that this term may have
          // and if it is >= smallestCount, then flag for refinement
          long maxCount = sfc.count;
          for (int shardNum=0; shardNum<rb.shards.length; shardNum++) {
            OpenBitSet obs = dff.counted[shardNum];
            if (!obs.get(sfc.termNum)) {
              // if missing from this shard, add the max it could be
              maxCount += dff.maxPossible(sfc,shardNum);
            }
          }
          if (maxCount >= smallestCount) {
            // TODO: on a tie, we could check the term values
            needRefinement = true;
          }
        }


        if (needRefinement) {
          // add a query for each shard missing the term that needs refinement
          for (int shardNum=0; shardNum<rb.shards.length; shardNum++) {
            OpenBitSet obs = dff.counted[shardNum];
            if (!obs.get(sfc.termNum) && dff.maxPossible(sfc,shardNum)>0) {
              dff.needRefinements = true;
              List<String> lst = dff._toRefine[shardNum];
              if (lst == null) {
                lst = dff._toRefine[shardNum] = new ArrayList<String>();
              }

View Full Code Here

    void add(int shardNum, NamedList shardCounts, int numRequested) {
      // shardCounts could be null if there was an exception
      int sz = shardCounts == null ? 0 : shardCounts.size();
      int numReceived = sz;


      OpenBitSet terms = new OpenBitSet(termNum+sz);


      long last = 0;
      for (int i=0; i<sz; i++) {
        String name = shardCounts.getName(i);
        long count = ((Number)shardCounts.getVal(i)).longValue();
        if (name == null) {
          missingCount += count;
          numReceived--;
        } else {
          ShardFacetCount sfc = counts.get(name);
          if (sfc == null) {
            sfc = new ShardFacetCount();
            sfc.name = name;
            sfc.indexed = ftype == null ? sfc.name : ftype.toInternal(sfc.name);
            sfc.termNum = termNum++;
            counts.put(name, sfc);
          }
          sfc.count += count;
          terms.fastSet(sfc.termNum);
          last = count;
        }
      }


      // the largest possible missing term is initialMincount if we received less

View Full Code Here

   * Inefficient base implementation.
   *
   * @see BitDocSet#getBits
   */
  public OpenBitSet getBits() {
    OpenBitSet bits = new OpenBitSet();
    for (DocIterator iter = iterator(); iter.hasNext();) {
      bits.set(iter.nextDoc());
    }
    return bits;
  };

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.lucene.util.OpenBitSet

com.browseengine.bobo.facets.data.MultiValueFacetDataCache

com.browseengine.bobo.facets.data.MultiValueWithWeightFacetDataCache

com.browseengine.bobo.facets.filter.BitSetFilter

com.browseengine.bobo.facets.filter.FacetOrFilter$FacetOrRandomAccessDocIdSet

com.browseengine.bobo.facets.filter.MultiValueORFacetFilter

com.browseengine.bobo.facets.range.ValueConverterBitSetBuilder

com.esri.gpt.catalog.lucene.AclFilter

com.esri.gpt.catalog.lucene.IsPartOfFilter

com.esri.gpt.catalog.lucene.SchemaFilter

com.esri.gpt.server.assertion.index.AsnFilter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.lucene.util.OpenBitSet

Performance Results

Related Classes of org.apache.lucene.util.OpenBitSet