Package co.cask.cdap.internal.app.runtime.batch

Examples of co.cask.cdap.internal.app.runtime.batch.MapReduceContextProvider

71
72
73
74
75
76
77
78
79
80
81
82
    DataSetInputSplit inputSplit = (DataSetInputSplit) split;

    Configuration conf = context.getConfiguration();
    // we don't currently allow datasets as the format between map and reduce stages, otherwise we'll have to
    // pass in the stage here instead of hardcoding mapper.
    MapReduceContextProvider contextProvider = new MapReduceContextProvider(context, MapReduceMetrics.TaskType.Mapper);
    BasicMapReduceContext mrContext = contextProvider.get();
    mrContext.getMetricsCollectionService().startAndWait();
    String dataSetName = getInputName(conf);
    BatchReadable<KEY, VALUE> inputDataset = (BatchReadable<KEY, VALUE>) mrContext.getDataSet(dataSetName);
    SplitReader<KEY, VALUE> splitReader = inputDataset.createSplitReader(inputSplit.getSplit());
View Full Code Here
50
51
52
53
54
55
56
57
58
59
60
61
  public RecordWriter<KEY, VALUE> getRecordWriter(final TaskAttemptContext context)
    throws IOException, InterruptedException {
    Configuration conf = context.getConfiguration();
    // we don't currently allow datasets as the format between map and reduce stages, otherwise we'll have to
    // pass in the stage here instead of hardcoding reducer.
    MapReduceContextProvider contextProvider = new MapReduceContextProvider(context, MapReduceMetrics.TaskType.Reducer);
    BasicMapReduceContext mrContext = contextProvider.get();
    mrContext.getMetricsCollectionService().startAndWait();
    @SuppressWarnings("unchecked")
    BatchWritable<KEY, VALUE> dataset = (BatchWritable<KEY, VALUE>) mrContext.getDataSet(getOutputDataSet(conf));

    // the record writer now owns the context and will close it
View Full Code Here
TOP

Related Classes of co.cask.cdap.internal.app.runtime.batch.MapReduceContextProvider

  • co.cask.cdap.common.conf.CConfiguration
  • co.cask.cdap.internal.app.runtime.batch.dataset.DataSetInputFormat
  • co.cask.cdap.internal.app.runtime.batch.dataset.DataSetOutputFormat
  • co.cask.cdap.internal.app.runtime.batch.distributed.DistributedMapReduceContextBuilder
  • co.cask.cdap.internal.app.runtime.batch.inmemory.InMemoryMapReduceContextBuilder
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.