Examples of ImportArcs

• org.archive.access.nutch.jobs.ImportArcs
  Ingests ARCs writing ARC Record parse as Nutch FetcherOutputFormat. FOF has five outputs:

  • crawl_fetch holds a fat CrawlDatum of all vitals including metadata. Its written below by our {@link WaxFetcherOutputFormat} (innutch by{@link FetcherOutputFormat}). Here is an example CD:

      Version: 4 Status: 5 (fetch_success) Fetch time: Wed Mar 15 12:38:49 PST 2006 Modified time: Wed Dec 31 16:00:00 PST 1969 Retries since fetch: 0 Retry interval: 0.0 days Score: 1.0 Signature: null Metadata: collection:test arcname:IAH-20060315203614-00000-debord arcoffset:5127  

  • crawl_parse has CrawlDatum of MD5s. Used making CrawlDB. Its obtained from above fat crawl_fetch CrawlDatum and written out as part of the parse output done by {@link WaxParseOutputFormat}. This latter class writes three files. This crawl_parse and both of the following parse_text and parse_data.
  • parse_text has text from parse.
  • parse_data has other metadata found by parse (Depends on parser). This is only input to linkdb. The html parser adds found out links here and content-type and discovered encoding as well as advertised encoding, etc.
  • cdx has a summary line for every record processed.

Examples of org.archive.access.nutch.jobs.ImportArcs