Examples of MRAsyncDiskService

org.apache.hadoop.mapreduce.util.MRAsyncDiskService
This class is a container of multiple thread pools, each for a volume, so that we can schedule async disk operations easily. Examples of async disk operations are deletion of files. We can move the files to a "toBeDeleted" folder before asychronously deleting it, to make sure the caller can run it faster. Users should not write files into the "toBeDeleted" folder, otherwise the files can be gone any time we restart the MRAsyncDiskService. This class also contains all operations that will be performed by the thread pools.
org.apache.hadoop.util.MRAsyncDiskService
This class is a container of multiple thread pools, each for a volume, so that we can schedule async disk operations easily. Examples of async disk operations are deletion of files. We can move the files to a "toBeDeleted" folder before asychronously deleting it, to make sure the caller can run it faster. Users should not write files into the "toBeDeleted" folder, otherwise the files can be gone any time we restart the MRAsyncDiskService. This class also contains all operations that will be performed by the thread pools.

Examples of org.apache.hadoop.util.MRAsyncDiskService

    assertTrue(fb.exists());
    assertTrue(fc.exists());
    assertTrue(fd.exists());
    
    // Create the asyncDiskService which will delete all contents inside SUBDIR
    MRAsyncDiskService service = new MRAsyncDiskService(
        localFileSystem, vols);
    
    // Make sure everything is cleaned up
    makeSureCleanedUp(vols, service);
  }

View Full Code Here

Examples of org.apache.hadoop.util.MRAsyncDiskService

             ", actual max reduce slots = " + actualMaxReduceSlots);


    // Check local disk, start async disk service, and clean up all
    // local directories.
    checkLocalDirs(getLocalDirsFromConf(this.fConf));
    asyncDiskService = new MRAsyncDiskService(FileSystem.getLocal(fConf),
        getLocalDirsFromConf(fConf), fConf);
    asyncDiskService.cleanupAllVolumes();
    DistributedCache.purgeCache(fConf, asyncDiskService);
    versionBeanName = VersionInfo.registerJMX("TaskTracker");

View Full Code Here

Examples of org.apache.hadoop.util.MRAsyncDiskService

    this.conf.set("mapred.job.tracker.http.address",
        infoBindAddress + ":" + this.infoPort);
    LOG.info("JobTracker webserver: " + this.infoServer.getPort());


    // start async disk service for asynchronous deletion service
    asyncDiskService = new MRAsyncDiskService(FileSystem.getLocal(jobConf),
        jobConf.getLocalDirs(), conf);


    while (!Thread.currentThread().isInterrupted()) {
      try {
        // if we haven't contacted the namenode go ahead and do it

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.