+ "Multiple --shard-url arguments can be specified, one for each desired shard. "
+ "If you are merging shards into a SolrCloud cluster, use --zk-host instead."));
Argument shardsArg = nonSolrCloud(solrClusterInfoGroup.addArgument("--shards")
.metavar("INTEGER")
.type(Integer.class).choices(new RangeArgumentChoice(1, Integer.MAX_VALUE))
.help("Number of output shards to generate."));
ArgumentGroup goLiveGroup = parser.addArgumentGroup("Go live arguments")
.description("Arguments for merging the shards that are built into a live Solr cluster. "
+ "Also see the Cluster arguments.");
Argument goLiveArg = goLiveGroup.addArgument("--go-live")
.action(Arguments.storeTrue())
.help("Allows you to optionally merge the final index shards into a live Solr cluster after they are built. "
+ "You can pass the ZooKeeper address with --zk-host and the relevant cluster information will be auto detected. "
+ nonSolrCloud("If you are not using a SolrCloud cluster, --shard-url arguments can be used to specify each SolrCore to merge "
+ "each shard into."));
Argument collectionArg = goLiveGroup.addArgument("--collection")
.metavar("STRING")
.help("The SolrCloud collection to merge shards into when using --go-live and --zk-host. Example: collection1");
Argument goLiveThreadsArg = goLiveGroup.addArgument("--go-live-threads")
.metavar("INTEGER")
.type(Integer.class)
.choices(new RangeArgumentChoice(1, Integer.MAX_VALUE))
.setDefault(1000)
.help("Tuning knob that indicates the maximum number of live merges to run in parallel at one time.");
ArgumentGroup optionalGroup = parser.addArgumentGroup("Optional arguments");
optionalGroup.addArgument("--help", "-help", "-h").help("Show this help message and exit")
.action(new HelpArgumentAction() {
@Override
public void run(ArgumentParser parser, Argument arg, Map<String, Object> attrs, String flag, Object value) throws ArgumentParserException {
parser.printHelp(new PrintWriter(System.out, true));
System.out.println();
System.out.print(ForkedToolRunnerHelpFormatter.getGenericCommandUsage());
System.out.println("Examples: \n\n" +
"# (Re)index a table in GoLive mode based on a local indexer config file\n" +
"hadoop --config /etc/hadoop/conf \\\n" +
" jar hbase-indexer-mr-*-job.jar \\\n" +
" --conf /etc/hbase/conf/hbase-site.xml \\\n" +
" -D 'mapred.child.java.opts=-Xmx500m' \\\n" +
" --hbase-indexer-file indexer.xml \\\n" +
" --zk-host 127.0.0.1/solr \\\n" +
" --collection collection1 \\\n" +
" --go-live \\\n" +
" --log4j src/test/resources/log4j.properties\n\n" +
"# (Re)index a table in GoLive mode using a local morphline-based indexer config file\n" +
"# Also include extra library jar file containing JSON tweet Java parser:\n" +
"hadoop --config /etc/hadoop/conf \\\n" +
" jar hbase-indexer-mr-*-job.jar \\\n" +
" --conf /etc/hbase/conf/hbase-site.xml \\\n" +
" --libjars /path/to/kite-morphlines-twitter-0.10.0.jar \\\n" +
" -D 'mapred.child.java.opts=-Xmx500m' \\\n" +
" --hbase-indexer-file src/test/resources/morphline_indexer_without_zk.xml \\\n" +
" --zk-host 127.0.0.1/solr \\\n" +
" --collection collection1 \\\n" +
" --go-live \\\n" +
" --morphline-file src/test/resources/morphlines.conf \\\n" +
" --output-dir hdfs://c2202.mycompany.com/user/$USER/test \\\n" +
" --overwrite-output-dir \\\n" +
" --log4j src/test/resources/log4j.properties\n\n" +
"# (Re)index a table in GoLive mode\n" +
"hadoop --config /etc/hadoop/conf \\\n" +
" jar hbase-indexer-mr-*-job.jar \\\n" +
" --conf /etc/hbase/conf/hbase-site.xml \\\n" +
" -D 'mapred.child.java.opts=-Xmx500m' \\\n" +
" --hbase-indexer-file indexer.xml \\\n" +
" --zk-host 127.0.0.1/solr \\\n" +
" --collection collection1 \\\n" +
" --go-live \\\n" +
" --log4j src/test/resources/log4j.properties\n\n" +
"# (Re)index a table with direct writes to SolrCloud\n" +
"hadoop --config /etc/hadoop/conf \\\n" +
" jar hbase-indexer-mr-*-job.jar \\\n" +
" --conf /etc/hbase/conf/hbase-site.xml \\\n" +
" -D 'mapred.child.java.opts=-Xmx500m' \\\n" +
" --hbase-indexer-file indexer.xml \\\n" +
" --zk-host 127.0.0.1/solr \\\n" +
" --collection collection1 \\\n" +
" --reducers 0 \\\n" +
" --log4j src/test/resources/log4j.properties\n\n" +
"# (Re)index a table based on a indexer config stored in ZK\n" +
"hadoop --config /etc/hadoop/conf \\\n" +
" jar hbase-indexer-mr-*-job.jar \\\n" +
" --conf /etc/hbase/conf/hbase-site.xml \\\n" +
" -D 'mapred.child.java.opts=-Xmx500m' \\\n" +
" --hbase-indexer-zk zk01 \\\n" +
" --hbase-indexer-name docindexer \\\n" +
" --go-live \\\n" +
" --log4j src/test/resources/log4j.properties\n\n");
throw new FoundHelpArgument(); // Trick to prevent processing of any remaining arguments
}
});
Argument outputDirArg = optionalGroup.addArgument("--output-dir")
.metavar("HDFS_URI")
.type(new PathArgumentType(conf) {
@Override
public Path convert(ArgumentParser parser, Argument arg, String value)
throws ArgumentParserException {
Path path = super.convert(parser, arg, value);
if ("hdfs".equals(path.toUri().getScheme())
&& path.toUri().getAuthority() == null) {
// TODO: consider defaulting to hadoop's
// fs.default.name here or in
// SolrRecordWriter.createEmbeddedSolrServer()
throw new ArgumentParserException("Missing authority in path URI: "
+ path, parser);
}
return path;
}
}.verifyHasScheme().verifyIsAbsolute().verifyCanWriteParent())
.help("HDFS directory to write Solr indexes to. Inside there one output directory per shard will be generated. "
+ "Example: hdfs://c2202.mycompany.com/user/$USER/test");
Argument overwriteOutputDirArg = optionalGroup.addArgument("--overwrite-output-dir")
.action(Arguments.storeTrue())
.help("Overwrite the directory specified by --output-dir if it already exists. Using this parameter will result in " +
"the output directory being recursively deleted at job startup.");
Argument morphlineFileArg = optionalGroup.addArgument("--morphline-file")
.metavar("FILE")
.type(new FileArgumentType().verifyExists().verifyIsFile().verifyCanRead())
.help("Relative or absolute path to a local config file that contains one or more morphlines. " +
"The file must be UTF-8 encoded. The file will be uploaded to each MR task. " +
"If supplied, this overrides the value from the --hbase-indexer-* options. " +
"Example: /path/to/morphlines.conf");
Argument morphlineIdArg = optionalGroup.addArgument("--morphline-id")
.metavar("STRING")
.type(String.class)
.help("The identifier of the morphline that shall be executed within the morphline config file, " +
"e.g. specified by --morphline-file. If the --morphline-id option is ommitted the first (i.e. " +
"top-most) morphline within the config file is used. If supplied, this overrides the value " +
"from the --hbase-indexer-* options. Example: morphline1 ");
Argument solrHomeDirArg = nonSolrCloud(optionalGroup.addArgument("--solr-home-dir")
.metavar("DIR")
.type(new FileArgumentType() {
@Override
public File convert(ArgumentParser parser, Argument arg, String value)
throws ArgumentParserException {
File solrHomeDir = super.convert(parser, arg, value);
File solrConfigFile = new File(new File(solrHomeDir, "conf"),
"solrconfig.xml");
new FileArgumentType().verifyExists().verifyIsFile().verifyCanRead()
.convert(parser, arg, solrConfigFile.getPath());
return solrHomeDir;
}
}.verifyIsDirectory().verifyCanRead())
.required(false)
.help("Relative or absolute path to a local dir containing Solr conf/ dir and in particular "
+ "conf/solrconfig.xml and optionally also lib/ dir. This directory will be uploaded to each MR task. "
+ "Example: src/test/resources/solr/minimr"));
Argument updateConflictResolverArg = optionalGroup.addArgument("--update-conflict-resolver")
.metavar("FQCN")
.type(String.class)
.setDefault(RetainMostRecentUpdateConflictResolver.class.getName())
.help("Fully qualified class name of a Java class that implements the UpdateConflictResolver interface. "
+ "This enables deduplication and ordering of a series of document updates for the same unique document "
+ "key. For example, a MapReduce batch job might index multiple files in the same job where some of the "
+ "files contain old and new versions of the very same document, using the same unique document key.\n"
+ "Typically, implementations of this interface forbid collisions by throwing an exception, or ignore all but "
+ "the most recent document version, or, in the general case, order colliding updates ascending from least "
+ "recent to most recent (partial) update. The caller of this interface (i.e. the Hadoop Reducer) will then "
+ "apply the updates to Solr in the order returned by the orderUpdates() method.\n"
+ "The default RetainMostRecentUpdateConflictResolver implementation ignores all but the most recent document "
+ "version, based on a configurable numeric Solr field, which defaults to the file_last_modified timestamp");
Argument reducersArg = optionalGroup.addArgument("--reducers")
.metavar("INTEGER")
.type(Integer.class)
.choices(new RangeArgumentChoice(-2, Integer.MAX_VALUE))
// TODO: also support X% syntax where X is an integer
.setDefault(-1)
.help("Tuning knob that indicates the number of reducers to index into. "
+ "0 indicates that no reducers should be used, and documents should be sent directly from the mapper tasks to live Solr servers. "
+ "-1 indicates use all reduce slots available on the cluster. "
+ "-2 indicates use one reducer per output shard, which disables the mtree merge MR algorithm. "
+ "The mtree merge MR algorithm improves scalability by spreading load "
+ "(in particular CPU load) among a number of parallel reducers that can be much larger than the number "
+ "of solr shards expected by the user. It can be seen as an extension of concurrent lucene merges "
+ "and tiered lucene merges to the clustered case. The subsequent mapper-only phase "
+ "merges the output of said large number of reducers to the number of shards expected by the user, "
+ "again by utilizing more available parallelism on the cluster.");
Argument fanoutArg = optionalGroup.addArgument("--fanout")
.metavar("INTEGER")
.type(Integer.class)
.choices(new RangeArgumentChoice(2, Integer.MAX_VALUE))
.setDefault(Integer.MAX_VALUE)
.help(FeatureControl.SUPPRESS);
Argument maxSegmentsArg = optionalGroup.addArgument("--max-segments")
.metavar("INTEGER")
.type(Integer.class)
.choices(new RangeArgumentChoice(1, Integer.MAX_VALUE))
.setDefault(1)
.help("Tuning knob that indicates the maximum number of segments to be contained on output in the index of "
+ "each reducer shard. After a reducer has built its output index it applies a merge policy to merge segments "
+ "until there are <= maxSegments lucene segments left in this index. "
+ "Merging segments involves reading and rewriting all data in all these segment files, "