Examples of Sharder

com.google.appengine.tools.mapreduce.Sharder
Used to determine which shard an item belongs to. This is used when emitting data from Map to specify which reduce shard it should go to. The only criteria that is required is that the same key always map to the same shard.
com.ngdata.hbaseindexer.indexer.Sharder
The Sharder interface. The sharder determines to which shard a document should be sent. Implementations should have a constructor that takes a single int argument that defines the number of shards.

Examples of com.google.appengine.tools.mapreduce.Sharder

      int shardCount) {
    List<ListMultimap<Bytes, V>> out = Lists.newArrayListWithCapacity(shardCount);
    for (int i = 0; i < shardCount; i++) {
      out.add(ArrayListMultimap.<Bytes, V>create());
    }
    Sharder sharder = new HashingSharder(shardCount);
    for (KeyValue<Bytes, V> pair : in) {
      out.get(sharder.getShardForKey(ByteBuffer.wrap(pair.getKey().bytes)))
          .put(pair.getKey(), pair.getValue());
    }
    return out;
  }

View Full Code Here

Examples of com.ngdata.hbaseindexer.indexer.Sharder

                DirectSolrInputDocumentWriter writer = createCloudSolrWriter(context, indexConnectionParams);
                solrDocWriter = wrapInBufferedWriter(context, writer);
                return Indexer.createIndexer(indexName, indexerConf, tableName, mapper, null, null, solrDocWriter);
            } else if (solrMode.equals("classic")) {
                DirectSolrClassicInputDocumentWriter classicSolrWriter = createClassicSolrWriter(context, indexConnectionParams);
                Sharder sharder = createSharder(indexConnectionParams, classicSolrWriter.getNumServers());
                solrDocWriter = wrapInBufferedWriter(context, classicSolrWriter);
                return Indexer.createIndexer(indexName, indexerConf, tableName, mapper, null, sharder, solrDocWriter);
            } else {
                throw new RuntimeException("Only 'cloud' and 'classic' are valid values for solr.mode, but got " + solrMode);
            }

View Full Code Here

Examples of com.ngdata.hbaseindexer.indexer.Sharder

            IndexerComponentFactory factory = IndexerComponentFactoryUtil.getComponentFactory(indexerDef.getIndexerComponentFactory(), new ByteArrayInputStream(indexerDef.getConfiguration()), indexerDef.getConnectionParams());
            IndexerConf indexerConf = factory.createIndexerConf();


            ResultToSolrMapper mapper = factory.createMapper(indexerDef.getName());


            Sharder sharder = null;
            SolrInputDocumentWriter solrWriter;
            PoolingClientConnectionManager connectionManager = null;


            if (indexerDef.getConnectionType() == null || indexerDef.getConnectionType().equals("solr")) {
                Map<String, String> connectionParams = indexerDef.getConnectionParams();

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.