Examples of cascading.property.ConfigDef

cascading.property.ConfigDef
The ConfigDef class allows for the creation of a configuration properties template to be applied to an existing properties configuration set.
There are three property modes, {@link Mode#DEFAULT}, {@link Mode#REPLACE}, and {@link Mode#UPDATE}.
- A DEFAULT property is only applied if there is no existing value in the property set.
- A REPLACE property is always applied overriding any previous values.
- An UPDATE property is always applied to an existing property. Usually when the property key represent a list of values.

   */
  @Override
  public ConfigDef getConfigDef()
    {
    if( configDef == null )
      configDef = new ConfigDef();


    return configDef;
    }

View Full Code Here

   */
  @Override
  public ConfigDef getStepConfigDef()
    {
    if( stepConfigDef == null )
      stepConfigDef = new ConfigDef();


    return stepConfigDef;
    }

View Full Code Here

   * @return an instance of ConfigDef
   */
  public ConfigDef getConfigDef()
    {
    if( configDef == null )
      configDef = new ConfigDef();


    return configDef;
    }

View Full Code Here

   */
  @Override
  public ConfigDef getStepConfigDef()
    {
    if( processConfigDef == null )
      processConfigDef = new ConfigDef();


    return processConfigDef;
    }

View Full Code Here

    smallPipe = new Each(smallPipe, new Fields(keyBytesField), new GetIndices(HashFunctionFactory.DEFAULT_HASH_FACTORY), new Fields("split", "index", "hash_num"));
    smallPipe = new Each(smallPipe, new Fields("split", "index", "hash_num"), new Unique.FilterPartialDuplicates());
    smallPipe = new GroupBy(smallPipe, new Fields("split"));
    smallPipe = new Every(smallPipe, new Fields("index", "hash_num"), new CreateBloomFilterFromIndices(), Fields.ALL);


    ConfigDef bloomDef = smallPipe.getStepConfigDef();
    bloomDef.setProperty(BloomProps.BLOOM_FILTER_PARTS_DIR, bloomPartsDir);
    bloomDef.setProperty(BloomProps.BLOOM_KEYS_COUNTS_DIR, approxCountPartsDir);
    bloomDef.setProperty(BloomProps.TARGET_BLOOM_FILTER_ID, bloomFilterID);


    setTails(smallPipe);
  }

View Full Code Here

    filterPipe = new NaiveMerge(largePipe.getName(), largePipe, smallPipe);


    // Load the bloom filter into memory and apply it to the LHS.
    filterPipe = new Each(filterPipe, largeJoinFields, new BloomJoinFilter(bloomJobID, false));


    ConfigDef config = filterPipe.getStepConfigDef();  // tell BloomAssemblyStrategy which bloom filter to expect
    config.setProperty(BloomProps.SOURCE_BLOOM_FILTER_ID, bloomJobID);
    config.setProperty(BloomProps.REQUIRED_BLOOM_FILTER_PATH, bloomFinalFilter);
    return filterPipe;
  }

View Full Code Here

TOP

Related Classes of cascading.property.ConfigDef

cascading.pipe.Pipe

cascading.tap.Tap

com.liveramp.cascading_ext.assembly.BloomAssembly

com.liveramp.cascading_ext.assembly.CreateBloomFilter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.