Examples of it.unimi.dsi.mg4j.index.Index

it.unimi.dsi.mg4j.index.Index
An abstract representation of an index.
Concrete subclasses of this class represent abstract index access information: for instance, the basename or IP address/port, flags, etc. It allows to build easily {@linkplain IndexReader index readers} over the index:in turn, index readers provide {@linkplain it.unimi.dsi.mg4j.search.DocumentIterator document iterators}.
In principle, this class should just contain methods declarations, and attributes for all data that is common to any form of index. Note that we use an abstract class, rather than an interface, because interfaces do not allow to declare attributes.
This class provide static factory methods (e.g., {@link #getInstance(CharSequence)}) that return an index given a suitable URI string. If the scheme part is mg4j, then the URI is assumed to point at a remote index. Otherwise, it is assumed to be the basename of a local index. In both cases, a query part introduced by ? can specify additional parameters (key=value pairs separated by ;). For instance, the URI example?inmemory=1 will load the index with basename example, caching its content in core memory. Please have a look at constants in {@link Index.UriKeys} (and analogous enums in subclasses) for additional parameters.
Thread safety

Indices are a natural candidate for multithreaded access. An instance of this class must be thread safe as long as external data structures provided to its constructors are. For instance, the tool {@link it.unimi.dsi.mg4j.tool.IndexBuilder} generatesa {@linkplain StringMaps#synchronize(PrefixMap) synchronized} {@link ImmutableExternalPrefixMap}so that by default the resulting index is thread safe.
For instance, a {@link it.unimi.dsi.mg4j.index.DiskBasedIndex} requires a list ofterm offsets, term maps, etc. As long as all these data structures are thread safe, the same is true of the index. Data structures created by static factory methods such as {@link it.unimi.dsi.mg4j.index.DiskBasedIndex#getInstance(CharSequence)} are thread safe.
Note that {@link it.unimi.dsi.mg4j.index.IndexReader}s returned by {@link #getReader()}are not thread safe (even if the method {@link #getReader()} is). The logic behindthis arrangement is that you create as many reader as you need, and then {@link java.io.Closeable#close()} them. In a multithreadedenvironment, a pool of index readers can be created, and a custom {@link it.unimi.dsi.mg4j.query.nodes.QueryBuilderVisitor}can be used to build {@link it.unimi.dsi.mg4j.search.DocumentIterator}s using the given pool of readers. In this case readers are not closed, but rather reused.
Read-once load

Implementations of this class are strongly encouraged to offer read-once constructors and factory methods: property files and other data related to the index (but not to an {@link it.unimi.dsi.mg4j.index.IndexReader}should be read exactly once, and sequentially. This feature is very useful when {@linkplain it.unimi.dsi.mg4j.tool.Combine combining indices}. @author Paolo Boldi @author Sebastiano Vigna @since 0.9

          weight = Double.parseDouble( basenameWeight[ i ].substring( split + 1 ) );
        }
        catch( NumberFormatException e ) {}
      }
        
      final Index index;


      if ( split == -1 || basenameWeight[ i ].startsWith("mg4j://") ) {
        index = Index.getInstance( basenameWeight[ i ], true, loadSizes );
        index2Weight.put( index, 1 );
      }

View Full Code Here

      throw new QueryBuilderVisitorException( e );
    }
  }
    
  public DocumentIterator visit( Range node ) throws QueryBuilderVisitorException {
    final Index index = curr.top();
    if ( ! index.hasPayloads ) throw new IllegalStateException( "Index " + index + " does not have payloads" );
    try {
      final Object parser = index2Parser.containsKey( index ) ? index2Parser.get( index ) : index.payload;
      final Method method = parser.getClass().getMethod( "parse", String.class );
      final Payload left = index.payload.copy(), right = index.payload.copy();
      if ( node.left != null ) left.set( method.invoke( parser, node.left.toString() ) );
      if ( node.right != null ) right.set( method.invoke( parser, node.right.toString() ) );
      return PayloadPredicateDocumentIterator.getInstance( index.documents( 0 ), 
          index.payload.rangeFilter( node.left == null ? null : left, node.right == null ? null : right ) ).weight( weight() );
    }
    catch( InvocationTargetException e ) {
      throw new QueryBuilderVisitorException( e.getCause() );
    }

View Full Code Here

    return result;
  }


  public boolean visitPre( final Select node ) throws QueryBuilderVisitorException {
    if ( indexMap == null ) throw new IllegalArgumentException( "You cannot use Select nodes without an index map" );
    final Index index = indexMap.get( node.index.toString() );
    if ( index == null ) throw new NoSuchElementException( "The selected index (" + node.index + ")" + " does not appear in the index map (" + indexMap + ")" ); 
    curr.push( indexMap.get( node.index.toString() ) );
    return true;
  }

View Full Code Here


  public DocumentIterator visitPost( final Remap node, final DocumentIterator subNode ) {
    if ( indexMap == null ) throw new IllegalArgumentException( "You cannot use Remap nodes without an index map" );
    final Reference2ReferenceArrayMap<Index, Index> indexInverseRemapping = new Reference2ReferenceArrayMap<Index, Index>( node.indexInverseRemapping.size() );
    for( Map.Entry<String,String> e: node.indexInverseRemapping.entrySet() ) {
      final Index externalIndex = indexMap.get( e.getKey() );
      final Index internalIndex = indexMap.get( e.getValue() );
      if ( internalIndex == null ) throw new NoSuchElementException( "The internal index \"" + e.getValue() + "\" does not appear in the index map (" + indexMap + ")" ); 
      if ( externalIndex == null ) throw new NoSuchElementException( "The external index \"" + e.getKey() + "\" does not appear in the index map (" + indexMap + ")" ); 
      indexInverseRemapping.put( externalIndex, internalIndex );
    }
    return new RemappingDocumentIterator( subNode, indexInverseRemapping );

View Full Code Here

                resultItem.uri = new MutableString( "./Item?doc=" ).append( resultItem.doc ).append( "&m=" ).append( urlEncodedMimeType );
              }
              
              MutableString text = new MutableString();
              for( Iterator<Index> j = indexMap.values().iterator(); j.hasNext(); ) {
                final Index index = j.next();
                selectedInterval = dsi.info.get( index );
                if ( selectedInterval != null )
                  text.append( "<p>" ).append( index.field ).append( ": " ).append( Arrays.asList( selectedInterval ) );
                LOGGER.debug( index.field + ": " + ( selectedInterval == null ? null : Arrays.asList( selectedInterval ) ) ); 
              }

View Full Code Here

      totalFrequency += ( frequency[ usedIndex[ k ] ] = indexIterator[ usedIndex[ k ] ].frequency() );


    if ( ! metadataOnly ) {
      int currIndex, numPrevDocs = 0, currDoc, count;
      OutputBitStream obs;
      Index i;
      IndexIterator ii;


      if ( p != 0 ) variableQuantumIndexWriter.newInvertedList(totalFrequency, p, predictedSize, predictedLengthNumBits ); 
      else indexWriter.newInvertedList();

View Full Code Here


      indexWriter.writeFrequency( totalFrequency );


      int currDoc = -1, count; 
      OutputBitStream obs;
      Index i;
      IndexIterator ir;


      while( ! documentQueue.isEmpty() ) {
        // We extract the smallest document pointer, and enqueue it in the new index.
        if ( currDoc == doc[ currIndex = documentQueue.first() ] ) throw new IllegalStateException( "The indices to be merged contain document " + currDoc + " at least twice (once in index " + inputBasename[ lastIndex ] + " and once in index " + inputBasename[ currIndex ] + ")" );

View Full Code Here

     * maxCount need not be updated, as it is already initialised to
     * the maximum over all indices. */
    int currIndex, prevDoc = -1, currDoc, count;
    int temp[];
    OutputBitStream obs;
    Index i;
    IndexIterator ii;
  
    // Note that the total frequency can be computed only during the merge.
    for( int k = numUsedIndices; k-- != 0; ) {
      currIndex = usedIndex[ k ];

View Full Code Here

  }
  
  public Boolean visit( final IndexIterator indexIterator ) throws IOException {
    // TODO: the second condition should be checked elsewhere, maybe...
    if ( indexIterator.frequency() > 0 && indexIterator.index().hasCounts) { // We skip empty iterators and indices without counts 
      final Index index = indexIterator.index();
      final String term = indexIterator.term();
      
      if ( term == null ) throw new NullPointerException( "This visitor needs a non-null term for each index iterator of nonzero frequency" );
      
      if ( ! term2Id.containsKey( term ) ) term2Id.put( term, term2Id.size() );

View Full Code Here

    final DataOutputStream outputStream = new DataOutputStream( socket.getOutputStream() );
    outputStream.writeByte( GET_INDEX );
    outputStream.writeBoolean( randomAccess );
    outputStream.writeBoolean( documentSizes );
    outputStream.flush();
    Index index = (Index)BinIO.loadObject( socket.getInputStream() );
    socket.close();
    LOGGER.debug( "Index at " + socket + " downloaded: " + index );
    return index;
  }

View Full Code Here

0 1

TOP

Related Classes of it.unimi.dsi.mg4j.index.Index

it.unimi.dsi.mg4j.examples.RunQuery

it.unimi.dsi.mg4j.index.cluster.DocumentalConcatenatedClusterDocumentIteratorTest

it.unimi.dsi.mg4j.index.MultiTermIndexIteratorTest

it.unimi.dsi.mg4j.index.remote.IndexServer

it.unimi.dsi.mg4j.query.Query

it.unimi.dsi.mg4j.query.QueryServlet

it.unimi.dsi.mg4j.search.BitStreamIndexReaderTest

it.unimi.dsi.mg4j.search.DocumentIteratorBuilderVisitor

it.unimi.dsi.mg4j.search.RemappingDocumentIterator

it.unimi.dsi.mg4j.search.score.AbstractWeightedScorer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of it.unimi.dsi.mg4j.index.Index

Thread safety

Read-once load

Related Classes of it.unimi.dsi.mg4j.index.Index