List of startNewPipeline() Examples

Examples of startNewPipeline()

com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()
Start a new Pipeline by specifying the root job and its arguments This version of the method is for root jobs that take zero arguments. @param jobInstance A job instance to use as the root job of the Pipeline @param settings Optional one or more {@code JobSetting} @return The pipeline handle. This String uniquely identifies the newlystarted Pipeline. It also uniquely identifies the root job of the Pipeline. The String may be used as an argument to {@link #getJobInfo(String)}, {@link #stopPipeline(String)}, and {@link #deletePipelineRecords(String,boolean,boolean)}

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()


    int entities = parseInt(req.getParameter("entities"));
    int bytesPerEntity = parseInt(req.getParameter("payloadBytesPerEntity"));
    int shardCount = parseInt(req.getParameter("shardCount"));
    PipelineService service = PipelineServiceFactory.newPipelineService();
    redirectToPipelineStatus(resp, service.startNewPipeline(
        new ChainedMapReduceJob(bucket, DATASTORE_TYPE, shardCount, entities, bytesPerEntity)));
  }
}

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

        .setReducer(ValueProjectionReducer.<String, Long>create())
        .setOutput(new CustomOutput())
        .setNumReducers(17);
    PipelineService pipelineService = PipelineServiceFactory.newPipelineService();
    MapReduceSettings mrSettings = new MapReduceSettings.Builder().build();
    String jobId = pipelineService.startNewPipeline(
        new MapReduceJob<>(mrSpecBuilder.build(), mrSettings));
    assertFalse(jobId.isEmpty());
    executeTasksUntilEmpty("default");
    JobInfo info = pipelineService.getJobInfo(jobId);
    assertEquals(State.COMPLETED_SUCCESSFULLY, info.getJobState());

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

      }
      return;
    }


    PipelineService service = PipelineServiceFactory.newPipelineService();
    redirectToPipelineStatus(resp, service.startNewPipeline(new BigQueryLoadJob(rowCount)));
  }


  private String getPipelineStatusUrl(String pipelineId) {
    return "/_ah/pipeline/status.html?root=" + pipelineId;
  }

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

    String bucket = getBucketParam(req);
    long start = getLongParam(req, "start", 0);
    long limit = getLongParam(req, "limit", 100 * 1000 * 1000);
    int shards = Math.max(1, Math.min(100, Ints.saturatedCast(getLongParam(req, "shards", 30))));
    PipelineService service = PipelineServiceFactory.newPipelineService();
    String pipelineId = service.startNewPipeline(new MyPipelineJob(bucket, start, limit, shards),
        new JobSetting.OnQueue(queue), new JobSetting.OnModule(module));
    resp.sendRedirect("/_ah/pipeline/status.html?root=" + pipelineId);
  }
}

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

      MapSettings settings) {
    if (settings.getWorkerQueueName() == null) {
      settings = new MapSettings.Builder(settings).setWorkerQueueName("default").build();
    }
    PipelineService pipelineService = PipelineServiceFactory.newPipelineService();
    return pipelineService.startNewPipeline(
        new MapJob<>(specification, settings), settings.toJobSettings());
  }


  @Override
  public Value<MapReduceResult<R>> run() {

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

    return futureCall(new DeletePipelineJob(getPipelineKey().getName()), settings);
  }


  public static void cleanup(List<GcsFilename> toDelete, JobSetting... settings) {
    PipelineService service = PipelineServiceFactory.newPipelineService();
    service.startNewPipeline(new CleanupPipelineJob(), toDelete, settings);
  }
}

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

      MapReduceSpecification<I, K, V, O, R> specification, MapReduceSettings settings) {
    if (settings.getWorkerQueueName() == null) {
      settings = new MapReduceSettings.Builder(settings).setWorkerQueueName("default").build();
    }
    PipelineService pipelineService = PipelineServiceFactory.newPipelineService();
    return pipelineService.startNewPipeline(
        new MapReduceJob<>(specification, settings), settings.toJobSettings());
  }


  /**
   * The pipeline job to execute the Map stage of the MapReduce. (For all shards)

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

      } else {
        log.info("Calling failed for " + jobId + ", status=" + jobState.getStatus());
        jobState.getController().failed(jobState.getStatus());
      }
      PipelineService pipeline = PipelineServiceFactory.newPipelineService();
      pipeline.startNewPipeline(
          new FinalizeShardedJob(jobId, jobState.getTotalTaskCount(), jobState.getStatus()));
    }
  }


  private IncrementalTaskState<T> getAndValidateTaskState(Transaction tx, String taskId,

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

      return false;
    }
    int taskCount = jobState.getTotalTaskCount();
    if (taskCount > 0) {
      PipelineService pipeline = PipelineServiceFactory.newPipelineService();
      pipeline.startNewPipeline(new DeleteShardedJob(jobId, taskCount));
    }
    final Key jobKey = ShardedJobStateImpl.ShardedJobSerializer.makeKey(jobId);
    RetryHelper.runWithRetries(callable(new Runnable() {
      @Override
      public void run() {

View Full Code Here

Examples of com.google.appengine.tools.pipeline.PipelineService.startNewPipeline()

  @SuppressWarnings("unchecked")
  @Test
  public void testExecute() {
    // setup service
    PipelineService mockPipeline = mock(PipelineService.class);
    when(mockPipeline.startNewPipeline(
        any(Job2.class), any(SelectStatement.class), any(StatementContext.class)
        )).thenReturn("key1").thenReturn("key2");
    
    UserServiceDelegate mockUserService = mock(UserServiceDelegate.class);
    when(mockUserService.getCurrentUser()).thenReturn(new User("testing@gmail.com", "google"));

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.