Examples of PartitionStrategy

com.cloudera.cdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

A {@code PartitionStrategy} is configured with one or more{@link FieldPartitioner}s upon creation. When a {@link Dataset} is configuredwith a partition strategy, we say that data is partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} which, in turn, produces a {@link PartitionKey}that is used by the dataset implementation to select the proper partition.

Users should use the inner {@link Builder} to create new instances.
@see FieldPartitioner @see PartitionKey @see DatasetDescriptor @see Dataset
org.kitesdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

When a {@link Dataset} is configuredwith a partition strategy, that data is considered partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} to determine which partition to write to.

You should use the inner {@link Builder} to create new instances.
@see DatasetDescriptor @see Dataset

Examples of org.kitesdk.data.PartitionStrategy

    Assert.assertEquals(strategy, PartitionStrategyParser.parseFromSchema(embedded));
  }


  @Test
  public void testReplaceEmbeddedPartitionStrategy() {
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .hash("username", 16)
        .identity("username", "u")
        .build();
    Schema original = new Schema.Parser().parse("{" +
        "  \"type\": \"record\"," +

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

        strategy, descriptor.getPartitionStrategy());
  }


  @Test
  public void testDescriptorValidationFails() {
    final PartitionStrategy missingStartField =
        new PartitionStrategy.Builder()
            .identity("p.latitude")
            .identity("p.longitude")
            .build();


    TestHelpers.assertThrows("Should complain that p is missing",
        IllegalStateException.class, new Runnable() {
          @Override
          public void run() {
            new DatasetDescriptor.Builder()
                .schema(schema)
                .partitionStrategy(missingStartField)
                .build();
          }
        });


    final PartitionStrategy missingLongField =
        new PartitionStrategy.Builder()
            .identity("position.latitude")
            .identity("position.long")
            .build();


    TestHelpers.assertThrows("Should complain that position.long is missing",
        IllegalStateException.class, new Runnable() {
          @Override
          public void run() {
            new DatasetDescriptor.Builder()
                .schema(schema)
                .partitionStrategy(missingLongField)
                .build();
          }
        });


    final PartitionStrategy badExpectedType =
        new PartitionStrategy.Builder()
            .identity("position.latitude")
            .year("position.longitude")
            .build();

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy


  }


  @Test
  public void testRefineIdentity() throws Exception {
      PartitionStrategy strategy = new PartitionStrategy.Builder()
              .identity("user_id")
              .build();


      DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
              .schemaUri("resource:standard_event.avsc")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    NOV_1 = new Marker.Builder()
        .add("year", 2013)
        .add("month", 11)
        .add("day", 1)
        .build();
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .year("timestamp")
        .month("timestamp")
        .day("timestamp")
        .build();
    comparator = new MarkerComparator(strategy);

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy


  @VisibleForTesting
  @SuppressWarnings("unchecked")
  Map<String, Predicate> minimizeFor(StorageKey key) {
    Map<String, Predicate> unsatisfied = Maps.newHashMap(constraints);
    PartitionStrategy strategy = key.getPartitionStrategy();
    Set<String> timeFields = Sets.newHashSet();
    int i = 0;
    for (FieldPartitioner fp : strategy.getFieldPartitioners()) {
      String partition = fp.getName();
      Predicate partitionPredicate = unsatisfied.get(partition);
      if (partitionPredicate != null && partitionPredicate.apply(key.get(i))) {
        unsatisfied.remove(partition);
        LOG.debug("removing " + partition + " satisfied by " + key.get(i));

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    this.conf = new Configuration();
    this.fileSystem = FileSystem.get(conf);
    this.testDirectory = new Path(Files.createTempDir().getAbsolutePath());
    this.repo = new FileSystemDatasetRepository(conf, testDirectory);


    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .hash("username", 2).build();
    FileSystemDataset<Object> users = (FileSystemDataset<Object>) repo.create(
        "ns", "users",
        new DatasetDescriptor.Builder()
            .schema(USER_SCHEMA)

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  public void testProvidedPartitioner() throws IOException {
    Schema user = SchemaBuilder.record("User").fields()
        .requiredString("username")
        .requiredString("email")
        .endRecord();
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .provided("version", "int")
        .build();
    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
        .schema(user)
        .partitionStrategy(strategy)

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("unchecked")
  public void testFromKey() {
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .year("timestamp")
        .month("timestamp")
        .day("timestamp")
        .build();

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("unchecked")
  public void testToKey() {
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .year("timestamp")
        .month("timestamp")
        .day("timestamp")
        .build();

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("unchecked")
  public void toDirNameIdentityWithSlashes() {
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .identity("name")
        .identity("address")
        .build();


    StorageKey key = new StorageKey(strategy);

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.