Examples of org.apache.nutch.fetcher.FetcherJob.fetch()

Class org.apache.nutch.fetcher.FetcherJob

Examples of org.apache.nutch.fetcher.FetcherJob.fetch()

org.apache.nutch.fetcher.FetcherJob.fetch()
Run fetcher. @param batchId batchId (obtained from Generator) or null to fetch all generated fetchlists @param threads number of threads per map task @param shouldResume @param numTasks number of fetching tasks (reducers). If set to < 1 then use the default,which is mapred.map.tasks. @return 0 on success @throws Exception

        LOG.info("Stopping at depth=" + i + " - no more URLs to fetch.");
        break;
      }
      boolean isParsing = getConf().getBoolean("fetcher.parse", false);
      start = System.currentTimeMillis();
      fetcher.fetch(batchId, threads, false, -1);  // fetch it
      delta = System.currentTimeMillis() - start;
      res.addTiming("fetch", i + "", delta);
      if (!isParsing) {
        start = System.currentTimeMillis();
        parseSegment.parse(batchId, false, false);    // parse it, if needed

View Full Code Here

        LOG.info("Stopping at depth=" + i + " - no more URLs to fetch.");
        break;
      }
      boolean isParsing = getConf().getBoolean("fetcher.parse", false);
      start = System.currentTimeMillis();
      fetcher.fetch(batchId, threads, false, -1);  // fetch it
      delta = System.currentTimeMillis() - start;
      res.addTiming("fetch", i + "", delta);
      if (!isParsing) {
        start = System.currentTimeMillis();
        parseBatch.parse(batchId, false, false);    // parse it, if needed

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.