Examples of it.unimi.dsi.mg4j.index.Index

it.unimi.dsi.mg4j.index.Index
An abstract representation of an index.
Concrete subclasses of this class represent abstract index access information: for instance, the basename or IP address/port, flags, etc. It allows to build easily {@linkplain IndexReader index readers} over the index:in turn, index readers provide {@linkplain it.unimi.dsi.mg4j.search.DocumentIterator document iterators}.
In principle, this class should just contain methods declarations, and attributes for all data that is common to any form of index. Note that we use an abstract class, rather than an interface, because interfaces do not allow to declare attributes.
This class provide static factory methods (e.g., {@link #getInstance(CharSequence)}) that return an index given a suitable URI string. If the scheme part is mg4j, then the URI is assumed to point at a remote index. Otherwise, it is assumed to be the basename of a local index. In both cases, a query part introduced by ? can specify additional parameters (key=value pairs separated by ;). For instance, the URI example?inmemory=1 will load the index with basename example, caching its content in core memory. Please have a look at constants in {@link Index.UriKeys} (and analogous enums in subclasses) for additional parameters.
Thread safety

Indices are a natural candidate for multithreaded access. An instance of this class must be thread safe as long as external data structures provided to its constructors are. For instance, the tool {@link it.unimi.dsi.mg4j.tool.IndexBuilder} generatesa {@linkplain StringMaps#synchronize(PrefixMap) synchronized} {@link ImmutableExternalPrefixMap}so that by default the resulting index is thread safe.
For instance, a {@link it.unimi.dsi.mg4j.index.DiskBasedIndex} requires a list ofterm offsets, term maps, etc. As long as all these data structures are thread safe, the same is true of the index. Data structures created by static factory methods such as {@link it.unimi.dsi.mg4j.index.DiskBasedIndex#getInstance(CharSequence)} are thread safe.
Note that {@link it.unimi.dsi.mg4j.index.IndexReader}s returned by {@link #getReader()}are not thread safe (even if the method {@link #getReader()} is). The logic behindthis arrangement is that you create as many reader as you need, and then {@link java.io.Closeable#close()} them. In a multithreadedenvironment, a pool of index readers can be created, and a custom {@link it.unimi.dsi.mg4j.query.nodes.QueryBuilderVisitor}can be used to build {@link it.unimi.dsi.mg4j.search.DocumentIterator}s using the given pool of readers. In this case readers are not closed, but rather reused.
Read-once load

Implementations of this class are strongly encouraged to offer read-once constructors and factory methods: property files and other data related to the index (but not to an {@link it.unimi.dsi.mg4j.index.IndexReader}should be read exactly once, and sequentially. This feature is very useful when {@linkplain it.unimi.dsi.mg4j.tool.Combine combining indices}. @author Paolo Boldi @author Sebastiano Vigna @since 0.9

  public boolean hasNext() {
    return documentIterator.hasNext();
  }


  private Index remapIndex( final Index index ) {
    final Index result = indexInverseRemapping.get(  index  );
    return result == null ? index : result;
  }

View Full Code Here

    return indices;
  }
  
  public IntervalIterator intervalIterator( final Index index ) throws IOException {
    if ( ! indices.contains( index ) ) return IntervalIterators.TRUE;
    final Index remappedIndex = remapIndex( index );


    IntervalIterator intervalIterator = currentIterators.get( index );
    if ( intervalIterator == null ) currentIterators.put( index, intervalIterator = documentIterator.intervalIterator( remappedIndex ) );
    return intervalIterator;
  }

View Full Code Here

 */


public class RunQuery {


  public static void main( String arg[] ) throws Exception {
    /** First we open our indices. The booleans tell that we want random access to
     * the inverted lists, and we are going to use document sizes (for scoring--see below). */
    final Index text = Index.getInstance( arg[ 0 ] + "-text", true, true );
    final Index title = Index.getInstance( arg[ 0 ] + "-title", true, true );


    /* We need a map mapping index names to actual indices. Its keyset will be used by the
     * parser to distinguish correct index names (e.g., "text:foo title:bar"), and the mapping
     * itself will be used when transforming a query into a document iterator. We use a handy
     * fastutil array-based constructor. */

View Full Code Here

   * 
   * @return the combined weighted score.
   */
  public double score() throws IOException {
    final double weight[] = this.currWeight;
    final Index index[] = this.currIndex;
    double result = 0;
    for( int i = n; i-- != 0; ) result += weight[ i ] * score( index[ i ] );
    return result;  
  }

View Full Code Here

                "A B C D E F F G G",
                "G A T H S K L J W L",
                "E S K D L J F K L S J D L S J D",
                "E B"
    ) ).run();
    Index index = DiskBasedIndex.getInstance( basename + "-text", true, true );


        /// String query = "A| B+C+G|W|S+J";
        DocumentIterator iterator = OrDocumentIterator.getInstance(
                index.documents("A"),
                MultiTermIndexIterator.getInstance(
                        index.documents("B"),
                        index.documents("C"),
                        index.documents("G")
                ),
                index.documents("W"),
                MultiTermIndexIterator.getInstance(
                        index.documents("S"),
                        index.documents("J")
                ));




        final int[] currDoc = new int[ 1 ];
        // A visitor invoking positionArray() on IndexIterators positioned on the current document.

View Full Code Here

  }
  
  public void testPaste() throws IOException, ConfigurationException, SecurityException, URISyntaxException, ClassNotFoundException, InstantiationException, IllegalAccessException, InvocationTargetException, NoSuchMethodException {
    
    
    Index index = Index.getInstance( basename );
    assertEquals( 2, index.documents( 0 ).frequency() );      
    assertEquals( 2, index.documents( 1 ).frequency() );
    assertEquals( 1, index.documents( 2 ).frequency() );
    assertEquals( 1, index.documents( 3 ).frequency() );
    
    SemiExternalGammaList frequencies = new SemiExternalGammaList( new InputBitStream( basename + "-mo" + DiskBasedIndex.FREQUENCIES_EXTENSION ), 1, 4 );


    assertEquals( 2, frequencies.getLong( 0 ) );      
    assertEquals( 2, frequencies.getLong( 1 ) );

View Full Code Here


  public void testSkipToEndOfList() throws ConfigurationException, SecurityException, IOException, URISyntaxException, ClassNotFoundException, InstantiationException, IllegalAccessException, InvocationTargetException, NoSuchMethodException {
    String basename = File.createTempFile( getClass().getSimpleName(), "test" ).getCanonicalPath();
    new IndexBuilder( basename, new StringArrayDocumentCollection( new String[] { "a", "a", "c" } ) ).run();


    Index index = DiskBasedIndex.getInstance( basename + "-text", true, true );
    IndexIterator indexIterator = index.documents( "a" );
    assertEquals( Integer.MAX_VALUE, indexIterator.skipTo( Integer.MAX_VALUE ) );
    indexIterator.dispose();


    indexIterator = index.documents( "a" );
    assertEquals( 0, indexIterator.skipTo( 0 ) );
    assertEquals( 1, indexIterator.skipTo( 1 ) );
    assertEquals( Integer.MAX_VALUE, indexIterator.skipTo( 2 ) );
    indexIterator.dispose();
}

View Full Code Here

    FileLinesCollection flc;
    flc = new FileLinesCollection( basename + "-cluster-0.terms", "ASCII" );
    BinIO.storeObject( new ShiftAddXorSignedStringMap( flc.iterator(), new MWHCFunction<CharSequence>( flc , TransformationStrategies.utf16() ) ), basename + "-cluster-0.termmap" );  
    flc = new FileLinesCollection( basename + "-cluster-1.terms", "ASCII" );
    BinIO.storeObject( new ShiftAddXorSignedStringMap( flc.iterator(), new MWHCFunction<CharSequence>( flc , TransformationStrategies.utf16() ) ), basename + "-cluster-1.termmap" );  
    Index index = Index.getInstance( basename + "-cluster" );
    assertEquals( Integer.MAX_VALUE, index.documents( "b" ).skipTo( 2 ) );
  }

View Full Code Here

0 1

TOP

Related Classes of it.unimi.dsi.mg4j.index.Index

it.unimi.dsi.mg4j.examples.RunQuery

it.unimi.dsi.mg4j.index.cluster.DocumentalConcatenatedClusterDocumentIteratorTest

it.unimi.dsi.mg4j.index.MultiTermIndexIteratorTest

it.unimi.dsi.mg4j.index.remote.IndexServer

it.unimi.dsi.mg4j.query.Query

it.unimi.dsi.mg4j.query.QueryServlet

it.unimi.dsi.mg4j.search.BitStreamIndexReaderTest

it.unimi.dsi.mg4j.search.DocumentIteratorBuilderVisitor

it.unimi.dsi.mg4j.search.RemappingDocumentIterator

it.unimi.dsi.mg4j.search.score.AbstractWeightedScorer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of it.unimi.dsi.mg4j.index.Index

Thread safety

Read-once load

Related Classes of it.unimi.dsi.mg4j.index.Index