Examples of org.apache.pig.builtin.BuildBloom

org.apache.pig.builtin.BuildBloom
Build a bloom filter for use later in Bloom. This UDF is intended to run in a group all job. For example: define bb BuildBloom('jenkins', '100', '0.1'); A = load 'foo' as (x, y); B = group A all; C = foreach B generate bb(A.x); store C into 'mybloom'; The bloom filter can be on multiple keys by passing more than one field (or the entire bag) to BuildBloom. The resulting file can then be used in a Bloom filter as: define bloom Bloom('mybloom'); A = load 'foo' as (x, y); B = load 'bar' as (z); C = filter B by bloom(z); D = join C by z, A by x; It uses {@link org.apache.hadoop.util.bloom.BloomFilter}.

    public void testBadHash() throws Exception {
        String size = "100";
        String numHash = "3";
        String hashFunc = "nosuchhash";
        try {
            BuildBloom bb = new BuildBloom(hashFunc, "fixed", size, numHash);
        } catch (RuntimeException re) {
            assertTrue(re.getMessage().contains("Unknown hash type"));
            throw re;
        }
    }

View Full Code Here

    @Test
    public void testFuncNames() throws Exception {
        String size = "100";
        String numHash = "3";
        String hashFunc = "JENKINS_HASH";
        BuildBloom bb = new BuildBloom(hashFunc, "fixed", size, numHash);
        assertEquals("org.apache.pig.builtin.BuildBloom$Initial",
                bb.getInitial());
        assertEquals("org.apache.pig.builtin.BuildBloom$Intermediate",
                bb.getIntermed());
        assertEquals("org.apache.pig.builtin.BuildBloom$Final",
                bb.getFinal());
    }

View Full Code Here

        String size = "100";
        String numHash = "3";
        String hashFunc = "nosuchhash";
        boolean caughtException = false;
        try {
            BuildBloom bb = new BuildBloom(hashFunc, "fixed", size, numHash);
        } catch (RuntimeException re) {
            assertTrue(re.getMessage().contains("Unknown hash type"));
            caughtException = true;
        }
        assertTrue(caughtException);

View Full Code Here

    @Test
    public void testFuncNames() throws Exception {
        String size = "100";
        String numHash = "3";
        String hashFunc = "JENKINS_HASH";
        BuildBloom bb = new BuildBloom(hashFunc, "fixed", size, numHash);
        assertEquals("org.apache.pig.builtin.BuildBloom$Initial",
            bb.getInitial());
        assertEquals("org.apache.pig.builtin.BuildBloom$Intermediate",
            bb.getIntermed());
        assertEquals("org.apache.pig.builtin.BuildBloom$Final",
            bb.getFinal());
    }

View Full Code Here

TOP

Related Classes of org.apache.pig.builtin.BuildBloom

org.apache.hadoop.util.bloom.BloomFilter

org.apache.hadoop.util.bloom.Key

org.apache.pig.data.DataBag

org.apache.pig.data.Tuple

org.apache.pig.impl.logicalLayer.schema.Schema

org.apache.pig.test.TestBloom

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.