Examples of GenericOptionsParser

org.apache.hadoop.util.GenericOptionsParser

GenericOptionsParser is a utility to parse command line arguments generic to the Hadoop framework. GenericOptionsParser recognizes several standarad command line arguments, enabling applications to easily specify a namenode, a jobtracker, additional configuration resources etc.

Generic Options

The supported generic options are:

 -conf <configuration file>     specify a configuration file -D <property=value>            use value for given property -fs <local|namenode:port>      specify a namenode -jt <local|jobtracker:port>    specify a job tracker -files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars>   specify comma separated jar files to include in the classpath. -archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is:

 bin/hadoop command [genericOptions] [commandOptions]

Generic command line arguments might modify Configuration objects, given to constructors.

The functionality is implemented using Commons CLI.

Examples:

 $ bin/hadoop dfs -fs darwin:8020 -ls /data list /data directory in dfs with namenode darwin:8020 $ bin/hadoop dfs -D fs.default.name=darwin:8020 -ls /data list /data directory in dfs with namenode darwin:8020 $ bin/hadoop dfs -conf hadoop-site.xml -ls /data list /data directory in dfs with conf specified in hadoop-site.xml $ bin/hadoop job -D mapred.job.tracker=darwin:50020 -submit job.xml submit a job to job tracker darwin:50020 $ bin/hadoop job -jt darwin:50020 -submit job.xml submit a job to job tracker darwin:50020 $ bin/hadoop job -jt local -submit job.xml submit a job to local runner $ bin/hadoop jar -libjars testlib.jar  -archives test.tgz -files file.txt inputjar args job submission with libjars, files and archives

@see Tool @see ToolRunner

Examples of org.apache.hadoop.util.GenericOptionsParser

  }


  @Override
  public int run(String[] args) throws Exception {
    setConf(HBaseConfiguration.create(getConf()));
    String[] otherArgs = new GenericOptionsParser(getConf(), args).getRemainingArgs();
    if (otherArgs.length < 2) {
      usage("Wrong number of arguments: " + otherArgs.length);
      return -1;
    }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

                    }
                }
                list.add(sb.toString());
            }


            new GenericOptionsParser(cfg, list.toArray(new String[list.size()]));
        } catch (Exception ex) {
            throw new EsHadoopIllegalStateException(ex);
        }
    }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

          }
        }
        list.add(sb.toString());
      }


      new GenericOptionsParser(cfg, list.toArray(new String[list.size()]));
    } catch (IOException ex) {
      throw new IllegalStateException(ex);
    }
  }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

      b.append(s);
      join = ",";
    }


    
    new GenericOptionsParser(conf, new String[]{"-libjars",b.toString()});
    System.out.println("tmpjars:"+b.toString()+"@@@@@"+conf.get("tmpjars"));




  }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser


    private LauncherOptions analyze() throws IOException, InterruptedException {
        LinkedList<String> copy = new LinkedList<String>(arguments);
        String applicationClassName = consumeApplicationClassName(copy);
        List<Path> libraryPaths = consumeLibraryPaths(copy);
        GenericOptionsParser genericOptions = processGenericOptions(copy);
        URLClassLoader applicationClassLoader = buildApplicationClassLoader(libraryPaths, applicationClassName);
        Class<? extends Tool> applicationClass = buildApplicationClass(applicationClassName);


        return new LauncherOptions(
                configuration,
                applicationClass,
                Arrays.asList(genericOptions.getRemainingArgs()),
                applicationClassLoader,
                applicationCacheFiles);
    }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

        return results;
    }


    private GenericOptionsParser processGenericOptions(LinkedList<String> rest) throws IOException {
        String[] args = rest.toArray(new String[rest.size()]);
        return new GenericOptionsParser(configuration, args);
    }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

    }
  }


  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if (otherArgs.length < 2) {
      System.err.println("Usage: wordcount <in> [<in>...] <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

    System.exit(ret);
  }


  @Override
  public int run(String[] args) throws Exception {
    String[] otherArgs = new GenericOptionsParser(getConf(), args).getRemainingArgs();
    Job job = createSubmittableJob(getConf(), otherArgs);
    if (job == null) return 1;
    return job.waitForCompletion(true) ? 0 : 1;
  }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

    return options;
  }


  private static CommandLine parseArguments(Configuration conf, Options options, String[] args)
      throws ParseException, IOException {
    GenericOptionsParser genParser = new GenericOptionsParser(conf, args);
    String[] remainingArgs = genParser.getRemainingArgs();
    CommandLineParser parser = new PosixParser();
    return parser.parse(options, remainingArgs);
  }

View Full Code Here

Examples of org.apache.hadoop.util.GenericOptionsParser

   * @param args  The command line parameters.
   * @throws Exception When running the job fails.
   */
  public static void main(String[] args) throws Exception {
    Configuration conf = HBaseConfiguration.create();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if (otherArgs.length < 2) {
      usage("Wrong number of arguments: " + otherArgs.length);
      System.exit(-1);
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.