Examples of eu.stratosphere.compiler.costs.DefaultCostEstimator

eu.stratosphere.compiler.costs.DefaultCostEstimator
A default cost estimator that has access to basic size and cardinality estimates.
This estimator works with actual estimates (as far as they are available) and falls back to setting relative costs, if no estimates are available. That way, the estimator makes sure that plans with different strategies are costed differently, also in the absence of estimates. The different relative costs in the absence of estimates represent this estimator's heuristic guidance towards certain strategies.
For robustness reasons, we always assume that the whole data is shipped during a repartition step. We deviate from the typical estimate of (n - 1) / n (with n being the number of nodes), because for a parallelism of 1, that would yield a shipping of zero bytes. While this is usually correct, the runtime scheduling may still choose to move tasks to different nodes, so that we do not know that no data is shipped.

   * it has no access to another cost estimator.
   * <p>
   * The address of the job manager (to obtain system characteristics) is determined via the global configuration.
   */
  public PactCompiler() {
    this(null, new DefaultCostEstimator());
  }

View Full Code Here

   * 
   * @param stats
   *        The statistics to be used to determine the input properties.
   */
  public PactCompiler(DataStatistics stats) {
    this(stats, new DefaultCostEstimator());
  }

View Full Code Here

  @Before
  public void setup() {
    InetSocketAddress dummyAddr = new InetSocketAddress("localhost", 12345);
    
    this.dataStats = new DataStatistics();
    this.withStatsCompiler = new PactCompiler(this.dataStats, new DefaultCostEstimator(), dummyAddr);
    this.withStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
    
    this.noStatsCompiler = new PactCompiler(null, new DefaultCostEstimator(), dummyAddr);
    this.noStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
    
    // create the instance type description
    InstanceType iType = InstanceTypeFactory.construct("standard", 6, 2, 4096, 100, 0);
    HardwareDescription hDesc = HardwareDescriptionFactory.construct(2, 4096 * 1024 * 1024, 2000 * 1024 * 1024);

View Full Code Here

  @Before
  public void setup() {
    InetSocketAddress dummyAddr = new InetSocketAddress("localhost", 12345);
    
    this.dataStats = new DataStatistics();
    this.withStatsCompiler = new PactCompiler(this.dataStats, new DefaultCostEstimator(), dummyAddr);
    this.withStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
    
    this.noStatsCompiler = new PactCompiler(null, new DefaultCostEstimator(), dummyAddr);
    this.noStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
    
    // create the instance type description
    InstanceType iType = InstanceTypeFactory.construct("standard", 6, 2, 4096, 100, 0);
    HardwareDescription hDesc = HardwareDescriptionFactory.construct(2, 4096 * 1024 * 1024, 2000 * 1024 * 1024);

View Full Code Here

    Preconditions.checkNotNull(config, "Configuration is null");
    this.configuration = config;
    configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, jobManagerAddress.getAddress().getHostAddress());
    configuration.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerAddress.getPort());
    
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator(), jobManagerAddress);
    
    //  Disable Local Execution when using a Client
    ContextEnvironment.disableLocalExecution();
  }

View Full Code Here

    if (port < 0) {
      throw new CompilerException("Cannot find port to job manager's RPC service in the global configuration.");
    }


    final InetSocketAddress jobManagerAddress = new InetSocketAddress(address, port);
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator(), jobManagerAddress);
    
    //  Disable Local Execution when using a Client
    ContextEnvironment.disableLocalExecution();
  }

View Full Code Here

TOP

Related Classes of eu.stratosphere.compiler.costs.DefaultCostEstimator

eu.stratosphere.client.program.Client

eu.stratosphere.compiler.PactCompiler

eu.stratosphere.pact.compiler.CompilerTestBase

eu.stratosphere.test.compiler.util.CompilerTestBase

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.