Examples of org.apache.nutch.indexer.solr.SolrDeleteDuplicates

org.apache.nutch.indexer.solr.SolrDeleteDuplicates
Utility class for deleting duplicate documents from a solr index. The algorithm goes like follows: Preparation:
1. Query the solr server for the number of documents (say, N)
2. Partition N among M map tasks. For example, if we have two map tasks the first map task will deal with solr documents from 0 - (N / 2 - 1) and the second will deal with documents from (N / 2) to (N - 1).
MapReduce:
- Map: Identity map where keys are digests and values are {@link SolrRecord}instances(which contain id, boost and timestamp)
- Reduce: After map, {@link SolrRecord}s with the same digest will be grouped together. Now, of these documents with the same digests, delete all of them except the one with the highest score (boost field). If two (or more) documents have the same score, then the document with the latest timestamp is kept. Again, every other is deleted from solr index.
Note that unlike {@link DeleteDuplicates} we assume that two documents ina solr index will never have the same URL. So this class only deals with documents with different URLs but the same digest.

        
        IndexingJob indexer = new IndexingJob(getConf());
        indexer.index(crawlDb, linkDb, 
                Arrays.asList(HadoopFSUtil.getPaths(fstats)));


        SolrDeleteDuplicates dedup = new SolrDeleteDuplicates();
        dedup.setConf(getConf());
        dedup.dedup(solrUrl);
      }
      
    } else {
      LOG.warn("No URLs to fetch - check your seed list and URL filters.");
    }

View Full Code Here

        // index, dedup & merge
        FileStatus[] fstats = fs.listStatus(segments, HadoopFSUtil.getPassDirectoriesFilter(fs));
        SolrIndexer indexer = new SolrIndexer(getConf());
        indexer.indexSolr(solrUrl, crawlDb, linkDb, 
          Arrays.asList(HadoopFSUtil.getPaths(fstats)));
        SolrDeleteDuplicates dedup = new SolrDeleteDuplicates();
        dedup.setConf(getConf());
        dedup.dedup(solrUrl);
      }
      
    } else {
      LOG.warn("No URLs to fetch - check your seed list and URL filters.");
    }

View Full Code Here

        // index, dedup & merge
        FileStatus[] fstats = fs.listStatus(segments, HadoopFSUtil.getPassDirectoriesFilter(fs));
        SolrIndexer indexer = new SolrIndexer(getConf());
        indexer.indexSolr(solrUrl, crawlDb, linkDb, 
          Arrays.asList(HadoopFSUtil.getPaths(fstats)));
        SolrDeleteDuplicates dedup = new SolrDeleteDuplicates();
        dedup.setConf(getConf());
        dedup.dedup(solrUrl);
      }
      
    } else {
      LOG.warn("No URLs to fetch - check your seed list and URL filters.");
    }

View Full Code Here

        // index, dedup & merge
        FileStatus[] fstats = fs.listStatus(segments, HadoopFSUtil.getPassDirectoriesFilter(fs));
        SolrIndexer indexer = new SolrIndexer(getConf());
        indexer.indexSolr(solrUrl, crawlDb, linkDb, 
          Arrays.asList(HadoopFSUtil.getPaths(fstats)));
        SolrDeleteDuplicates dedup = new SolrDeleteDuplicates();
        dedup.setConf(getConf());
        dedup.dedup(solrUrl);
      }
      
    } else {
      LOG.warn("No URLs to fetch - check your seed list and URL filters.");
    }

View Full Code Here

TOP

Related Classes of org.apache.nutch.indexer.solr.SolrDeleteDuplicates

org.apache.hadoop.conf.Configuration

org.apache.hadoop.io.Text

org.apache.hadoop.mapred.JobConf

org.apache.hadoop.mapreduce.Job

org.apache.nutch.crawl.Crawl

org.apache.nutch.util.NutchJob

org.apache.solr.client.solrj.impl.CommonsHttpSolrServer

org.apache.solr.client.solrj.request.UpdateRequest

org.apache.solr.client.solrj.response.QueryResponse

org.apache.solr.client.solrj.SolrQuery

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.