Examples of Paste

it.unimi.dsi.mg4j.tool.Paste
Pastes several indices.
Pasting is a very slow way of combining indices: we assume that not only documents, but also document occurrences might be scattered throughout several indices. When a document appears in several indices, its occurrences in a given index are combined. We have two possibilities:
- standard pasting: position lists are simply concatenated—it is responsibility of the caller to guarantee that they have been numbered in an increasing fashion; the sizes of the last input index are the sizes of the pasted index;
- incremental pasting: position lists are concatenated, but each list is renumbered by adding to all positions the sum of the sizes of the current document for all indices the precede the current one (this kind of pasting was the only one available before version 3.0).
Standard pasting is used, for instance, to paste the batches of a {@linkplain it.unimi.dsi.mg4j.document.DocumentFactory.FieldType#VIRTUAL virtual field}generated by {@link Scan}; the latter takes care of numbering positions correctly. If, however, you index parts of the same document collection on different machines using the same {@link VirtualDocumentResolver}, then the resulting indices for virtual fields will have all position starting from zero, and they will need an incremental pasting to be combined correctly.
Conceptually, this operation is equivalent to splitting a collection vertically: each document is divided into a fixed number n of consecutive segments (possibly of length 0), and a set of n indices is created using the k-th segment of all documents. Pasting the resulting indices will produce an index that is identical to the index generated by the original collection. The behaviour is analogous to that of the UN*X paste command if documents are single-line lists of words.
Note that in case every document appears at most in one index pasting is equivalent to {@linkplain it.unimi.dsi.mg4j.tool.Merge merging}. It is, however, significantly slower, as the presence of the same document in several lists makes it necessary to scan completely the inverted lists to be pasted to compute the frequency. To do so, an in-memory buffer is allocated. If an inverted list does not fit in the memory buffer, it is spilled on disk. Sizing correctly the buffer, and choosing a fast file system for the temporary directory can significantly affect performance.
Warning: incremental pasting is very memory-intensive, as a list of sizes must be loaded for each index. You can use the URI option succinctsizes=1 to load sizes in a succinct format, which will ease the problem. @author Sebastiano Vigna @since 1.0
models.Paste
palmed.edit.selection.Paste
This class implements an action to paste text replacing the current selection. @author Mathieu Champlon @version $Revision$ $Date$

Examples of it.unimi.dsi.mg4j.tool.Paste

    indexTitle = DiskBasedIndex.getInstance( titleBasename + "-text", true, true );
    indexFBody = DiskBasedIndex.getInstance( bodyFBasename + "-text", true, true );
    indexFTitle = DiskBasedIndex.getInstance( titleFBasename + "-text", true, true );
    indexBodyBis = DiskBasedIndex.getInstance( bodyBasenameBis + "-text", true, true );


    new Paste( basenameFComb, new String[] { bodyFBasename + "-text", titleFBasename + "-text" }, true, true, 1024, null, 1024, CompressionFlags.DEFAULT_STANDARD_INDEX, true, false, -1, -1, -1, 10 ).run();
    immutableExternalPrefixMap = new ImmutableExternalPrefixMap( new FileLinesCollection( basenameFComb + ".terms", "UTF-8" ) );
    simpleParser = new SimpleParser( index0.termProcessor );
  }

View Full Code Here

Examples of models.Paste

            final String actualLang = Languages.forRequestParameter(lang);
            final String sha1 = Hash.of(body, actualLang);


            Logger.debug("Got some code, sha1 is " + sha1);


            final Paste paste = Paste.findByHash(sha1);
            if (paste != null)
            {
                Logger.info("Found id " + paste.key + " for code with sha1 " + sha1);
                return ok(pastebinMacro.render(paste.key, body));
            }


            Logger.debug("Need to generate an id for code with sha1 " + sha1);


            return Api.post(body, actualLang, new Api.Call<Result>()
            {
                public Result call(String url, ImmutableMap<String, String> parameters)
                {
                    return async(WsUtils.post(url, parameters)
                            .map(new F.Function<WS.Response, Result>()
                            {
                                public Result apply(WS.Response response) throws Throwable
                                {
                                    final String responseBody = response.getBody();
                                    if (PastebinUrl.isUrl(responseBody))
                                    {
                                        final String pasteId = PastebinUrl.id(responseBody);


                                        final Paste p = new Paste();
                                        p.key = pasteId;
                                        p.hash = sha1;
                                        Paste.create(p);


                                        Logger.info("Created id " + pasteId + " for code with sha1 " + sha1);

View Full Code Here

Examples of palmed.edit.selection.Paste

    /**
     * {@inheritDoc}
     */
    public void paste( final InputStream stream )
    {
        execute( new Paste( selection_, stream ) );
    }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.