Examples of voldemort.client.protocol.admin.StreamingClient

voldemort.client.protocol.admin.StreamingClient

This class is a wrapper on the StreamingClient It serves the purpose of making streaming resilient to single node failures and making it highly available. The StreamingClient stops streaming when a voldemort node is down The streaming client interface would expect the data producer(eg: faust) to provide the failed list of nodes upfront. On a node failure the streaming client also provides the list of faulty nodes which can be used by the data producer to reinitialize the streaming session. Why did we choose this design: ==> The streaming session does not buffer the entries, it just keeps them on a network buffer, if a node fails during this time, the following flush() call will yield and generate an exception Only the data producer can resend those events, thereby re-initializing the session

        Props property = new Props();
        property.put("streaming.platform.bootstrapURL", bootstrapUrl);
        property.put("streaming.platform.max.failed.nodes", numFailedNodes);


        StreamingClientConfig config = new StreamingClientConfig(property);
        StreamingClient streamer = new StreamingClient(config);


        Callable<Integer> cpCallable = new CheckpointCallable();
        Callable<Integer> rbCallable = new RollbackCallable();
        List<String> stores = new ArrayList();


        stores.add(STORE_NAME);
        List<Integer> failedNodes = new ArrayList<Integer>();


        for(int i = 0; i < numFailedNodes; i++)
            failedNodes.add(i + 1);
        if(!wrapException) {
            streamer.initStreamingSessions(stores, cpCallable, rbCallable, true, failedNodes);
        }


        else {
            try {
                streamer.initStreamingSessions(stores, cpCallable, rbCallable, true, failedNodes);
            } catch(Exception e) {
                logger.error(e);
            }
        }
        return streamer;

Examples of voldemort.client.protocol.admin.StreamingClient

Related Classes of voldemort.client.protocol.admin.StreamingClient