Examples of PartitionStrategy

com.cloudera.cdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

A {@code PartitionStrategy} is configured with one or more{@link FieldPartitioner}s upon creation. When a {@link Dataset} is configuredwith a partition strategy, we say that data is partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} which, in turn, produces a {@link PartitionKey}that is used by the dataset implementation to select the proper partition.

Users should use the inner {@link Builder} to create new instances.
@see FieldPartitioner @see PartitionKey @see DatasetDescriptor @see Dataset
org.kitesdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

When a {@link Dataset} is configuredwith a partition strategy, that data is considered partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} to determine which partition to write to.

You should use the inner {@link Builder} to create new instances.
@see DatasetDescriptor @see Dataset

Examples of org.kitesdk.data.PartitionStrategy

    // convert the schema to Hive columns
    table.getSd().setCols(convertSchema(descriptor.getSchema()));


    // copy partitioning info
    if (descriptor.isPartitioned()) {
      PartitionStrategy ps = descriptor.getPartitionStrategy();
      table.getParameters().put(PARTITION_EXPRESSION_PROPERTY_NAME,
          Accessor.getDefault().toExpression(ps));
      table.setPartitionKeys(partitionColumns(ps, descriptor.getSchema()));
    }

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    }
    // Ensure schema is limited to keySchema's fields. The class may have more
    // fields
    // in the case that the entity is being used as a key.
    List<Field> fields = Lists.newArrayList();
    PartitionStrategy strategy = keySchema.getPartitionStrategy();
    for (Schema.Field field : keySchema.getAvroSchema().getFields()) {
      String sourceName = strategy.getPartitioner(field.name()).getSourceName();
      fields.add(copy(schemaField.getField(sourceName)));
    }
    Schema schema = Schema.createRecord(keySchema.getAvroSchema().getName(),
        keySchema.getAvroSchema().getDoc(), keySchema.getAvroSchema()
            .getNamespace(), keySchema.getAvroSchema().isError());

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  @SuppressWarnings("deprecation")
  @Test
  public void testNewPartitionIsVisibleToHive() throws Exception {
    final String NAME2 = "test2";


    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .hash("username", 2).build();


    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
        .schema(testSchema)
        .partitionStrategy(partitionStrategy)

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  @Test
  public void testMerge() throws Exception {
    final String NAME2 = "test2";
    final String NAME3 = "test3";


    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .hash("username", 2).build();


    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
        .schema(testSchema)
        .partitionStrategy(partitionStrategy)

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    NOV_1 = new Marker.Builder()
        .add("year", 2013)
        .add("month", 11)
        .add("day", 1)
        .build();
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .year("timestamp")
        .month("timestamp")
        .day("timestamp")
        .build();
    comparator = new MarkerComparator(strategy);

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    }


    @Override
    public void initialize() {
      if (key == null) {
        PartitionStrategy strategy = PartitionStrategyParser.parse(strategyString);
        Schema schema = new Schema.Parser().parse(schemaString);
        this.key = new AvroStorageKey(strategy, schema);
        this.accessor = DataModelUtil.accessor(type, schema);
      }
    }

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

        .from("created_at",
            new DateTime(2013, 1, 1, 0, 0, DateTimeZone.UTC).getMillis())
        .toBefore("created_at",
            new DateTime(2014, 1, 1, 0, 0, DateTimeZone.UTC).getMillis());


    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .hash("color", "hash", 50)
        .year("created_at").month("created_at").day("created_at")
        .identity("color")
        .build();

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

        c3.minimizeFor(key).keySet());
  }


  @Test
  public void testAlignedWithPartitionBoundaries() {
    PartitionStrategy hashStrategy = new PartitionStrategy.Builder()
        .hash("id", "bucket", 32)
        .build();
    PartitionStrategy withColor = new PartitionStrategy.Builder()
        .range("color", "blue", "green", "red")
        .year("timestamp")
        .month("timestamp")
        .day("timestamp")
        .identity("id")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    checkTestUsers(outputDataset, 10);
  }


  @Test
  public void testPartitionedSource() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
        "username", 2).build();


    Dataset<Record> inputDataset = repo.create("ns", "in", new DatasetDescriptor.Builder()
        .schema(USER_SCHEMA).partitionStrategy(partitionStrategy).build());
    Dataset<Record> outputDataset = repo.create("ns", "out", new DatasetDescriptor.Builder()

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    Assert.assertEquals(5, datasetSize(outputDataset));
  }


  @Test
  public void testPartitionedSourceAndTarget() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
        "username", 2).build();


    Dataset<Record> inputDataset = repo.create("ns", "in", new DatasetDescriptor.Builder()
        .schema(USER_SCHEMA).partitionStrategy(partitionStrategy).build());
    Dataset<Record> outputDataset = repo.create("ns", "out", new DatasetDescriptor.Builder()

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.