Examples of TRECDocumentCollection

it.unimi.dsi.mg4j.document.TRECDocumentCollection
A collection for the TREC GOV2 data set.
The documents are stored as a set of descriptors, representing the (possibly gzipped) file they are contained in and the start and stop position in that file. To manage descriptors later we rely on {@link SegmentedInputStream}.
To interpret a file, we read up to <DOC> and place a start marker there, we advance to the header and store the URI. An intermediate marker is placed at the end of the doc header tag and a stop marker just before </DOC>.
The resulting {@link SegmentedInputStream} has two segmentsper document. By using a {@link it.unimi.dsi.mg4j.document.CompositeDocumentFactory}, the first segment is parsed by a {@link it.unimi.dsi.mg4j.document.TRECHeaderDocumentFactory}, whereas the second segment is parsed by a user-provided factory—usually, an {@link it.unimi.dsi.mg4j.document.HtmlDocumentFactory}.
The collection provides both sequential access to all documents via the iterator and random access to a given document. However, the two operations are performed very differently as the sequential operation is much more efficient than calling {@link #document(int)} repeatedly. @author Alessio Orlandi @author Luca Natali
net.bpiwowar.mg4j.extensions.trec.TRECDocumentCollection
A collection for the TREC data set.

The documents are stored as a set of descriptors, representing the (possibly gzipped) files they are contained in and the start and stop position in that files. To manage descriptors later we rely on {@link SegmentedInputStream}.

To interpret a files, we read up to <DOC> and place a start marker there, we advance to the header and store the URI. An intermediate marker is placed at the end of the doc header tag and a stop marker just before </DOC>.

The collection provides both sequential access to all documents via the iterator and random access to a given document. However, the two operations are performed very differently as the sequential operation is much more performant than calling {@link #document(long)} repeatedly. @author Alessio Orlandi @author Luca Natali @author Benjamin Piwowarski

Examples of it.unimi.dsi.mg4j.document.TRECDocumentCollection

    OutputStream outputStream = new FileOutputStream(temp);
    IOUtils.copy(this.getClass().getResourceAsStream("testChar255.data"),
        outputStream);
    outputStream.close();


    TRECDocumentCollection collection = new TRECDocumentCollection(
        new String[] { temp.toString() },
        CompositeDocumentFactory
            .getFactory(new DocumentFactory[] {
                new TRECHeaderDocumentFactory(),
                new HtmlDocumentFactory( new String[] { "encoding=ISO-8859-1" } ) } ),
        4, // Very small, to induce fragmentation
        false);


    try {
      DocumentIterator iter = collection.iterator();
      Document d;
      while ((d = iter.nextDocument()) != null)
        d.title();
    } catch (IllegalStateException e) {
      assertTrue(false);

View Full Code Here

Examples of it.unimi.dsi.mg4j.document.TRECDocumentCollection

    IOUtils.copy( this.getClass().getResourceAsStream( "testContents.data" ), outputStream );
    outputStream.close();
    IOUtils.copy( this.getClass().getResourceAsStream( "testContentsAgain.data" ), outputStreamAgain );
    outputStreamAgain.close();


    TRECDocumentCollection collection = new TRECDocumentCollection(
        new String[] { temp.toString(), tempAgain.toString() },
        CompositeDocumentFactory
            .getFactory(new DocumentFactory[] {
                new TRECHeaderDocumentFactory(),
                new HtmlDocumentFactory( new String[] { "encoding=ISO-8859-1" } ) } ),
        4, // Very small, to induce fragmentation
        false);


    DocumentIterator iter = collection.iterator();
    Document d = null;


    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0001/", d.uri());
    assertEquals("GX001", d.title());


    final int textIndex = collection.factory().fieldIndex( "text" );
    
    assertEquals( "Line 1\n     The line 2!\n  Mamma\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    
    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0002/", d.uri());
    assertEquals("GX002", d.title());


    assertEquals( "Contents of this file reside on one line only\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );


    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0003/", d.uri());
    assertEquals("GX003", d.title());


    assertEquals( "Line 1\nLine 2\nLine 3\nLine 4\nLine 5\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );


    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0004/", d.uri());
    assertEquals("GX004", d.title());


    assertEquals( "New content 0\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    
    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0005/", d.uri());
    assertEquals("GX005", d.title());


    assertEquals( "New content 1\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );


    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0006/", d.uri());
    assertEquals("GX006", d.title());


    assertEquals( "New content 2\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );


    d = iter.nextDocument();
    assertNotNull(d);
    assertEquals("http://gx0007/", d.uri());
    assertEquals("GX007", d.title());


    assertEquals( "", IOUtils.toString( (Reader)d.content( textIndex ) ) );


    d = iter.nextDocument();
    assertNull(d);
    iter.close();
    
    d = collection.document( 0 );
    assertNotNull(d);
    assertEquals("http://gx0001/", d.uri());
    assertEquals("GX001", d.title());


    assertEquals( "Line 1\n     The line 2!\n  Mamma\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();
    
    d = collection.document( 1 );
    assertNotNull(d);
    assertEquals("http://gx0002/", d.uri());
    assertEquals("GX002", d.title());


    assertEquals( "Contents of this file reside on one line only\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();


    d = collection.document( 2 );
    assertNotNull(d);
    assertEquals("http://gx0003/", d.uri());
    assertEquals("GX003", d.title());


    assertEquals( "Line 1\nLine 2\nLine 3\nLine 4\nLine 5\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();


    d = collection.document( 3 );
    assertNotNull(d);
    assertEquals("http://gx0004/", d.uri());
    assertEquals("GX004", d.title());


    assertEquals( "New content 0\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();
    
    d = collection.document( 4 );
    assertNotNull(d);
    assertEquals("http://gx0005/", d.uri());
    assertEquals("GX005", d.title());


    assertEquals( "New content 1\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();


    d = collection.document( 5 );
    assertNotNull(d);
    assertEquals("http://gx0006/", d.uri());
    assertEquals("GX006", d.title());


    assertEquals( "New content 2\n", IOUtils.toString( (Reader)d.content( textIndex ) ) );
    d.close();


    d = collection.document( 6 );
    assertNotNull(d);
    assertEquals("http://gx0007/", d.uri());
    assertEquals("GX007", d.title());


    assertEquals( "", IOUtils.toString( (Reader)d.content( textIndex ) ) );

View Full Code Here

Examples of net.bpiwowar.mg4j.extensions.trec.TRECDocumentCollection

            case "trec":
                Properties properties = new Properties();
                properties.setProperty(PropertyBasedDocumentFactory.MetadataKeys.ENCODING, "UTF-8");
                final TRECDocumentFactory documentFactory = new TRECDocumentFactory(properties);


                collection = new TRECDocumentCollection(files,
                            documentFactory, SegmentedDocumentCollection.DEFAULT_BUFFER_SIZE, compression, metadataFile);
                break;


            case "warc/0.18":
                collection = new WARCDocumentCollection(files, SegmentedDocumentCollection.DEFAULT_BUFFER_SIZE, compression, metadataFile);

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.