Examples of EtlRequest

com.linkedin.camus.etl.kafka.common.EtlRequest
A class that represents the kafka pull request. The class is a container for topic, leaderId, partition, uri and offset. It is used in reading and writing the sequence files used for the extraction job. @author Richard Park

Examples of com.linkedin.camus.etl.kafka.common.EtlRequest

        Message message = null;


        while (true) {
            try {
                if (reader == null || !reader.hasNext()) {
                    EtlRequest request = (EtlRequest) split.popRequest();
                    if (request == null) {
                        return false;
                    }


                    if (maxPullHours > 0) {
                        endTimeStamp = 0;
                    }


                    key.set(request.getTopic(), request.getLeaderId(), request.getPartition(),
                            request.getOffset(), request.getOffset(), 0);
                    value = null;
                    log.info("\n\ntopic:" + request.getTopic() + " partition:"
                            + request.getPartition() + " beginOffset:" + request.getOffset()
                            + " estimatedLastOffset:" + request.getLastOffset());


                    statusMsg += statusMsg.length() > 0 ? "; " : "";
                    statusMsg += request.getTopic() + ":" + request.getLeaderId() + ":"
                            + request.getPartition();
                    context.setStatus(statusMsg);


                    if (reader != null) {
                        closeReader();
                    }
                    reader = new KafkaReader(context, request,
                            CamusJob.getKafkaTimeoutValue(mapperContext),
                            CamusJob.getKafkaBufferSize(mapperContext));


                    decoder = MessageDecoderFactory.createMessageDecoder(context, request.getTopic());
                }
                int count = 0;
                while (reader.getNext(key, msgValue, msgKey)) {
                    readBytes += key.getMessageSize();
                    count++;

View Full Code Here

Examples of com.linkedin.camus.etl.kafka.common.EtlRequest

        long earliestOffset = earliestOffsetResponse.offsets(
            topicAndPartition.topic(),
            topicAndPartition.partition())[0];
        
        //TODO: factor out kafka specific request functionality 
        CamusRequest etlRequest = new EtlRequest(context,
            topicAndPartition.topic(), Integer.toString(leader
                .getLeaderId()), topicAndPartition.partition(),
            leader.getUri());
        etlRequest.setLatestOffset(latestOffset);
        etlRequest.setEarliestOffset(earliestOffset);
        finalRequests.add(etlRequest);
      }
    }
    return finalRequests;
  }

View Full Code Here

Examples of com.linkedin.camus.etl.kafka.common.EtlRequest

        SequenceFile.Reader reader = new SequenceFile.Reader(fs,
            f.getPath(), context.getConfiguration());
        EtlKey key = new EtlKey();
        while (reader.next(key, NullWritable.get())) {
        //TODO: factor out kafka specific request functionality 
          CamusRequest request = new EtlRequest(context,
              key.getTopic(), key.getLeaderId(),
              key.getPartition());
          if (offsetKeysMap.containsKey(request)) {


            EtlKey oldKey = offsetKeysMap.get(request);

View Full Code Here

Examples of com.linkedin.camus.etl.kafka.common.EtlRequest


  @Override
  public void readFields(DataInput in) throws IOException {
    int size = in.readInt();
    for (int i = 0; i < size; i++) {
      CamusRequest r = new EtlRequest();
      r.readFields(in);
      requests.add(r);
      length += r.estimateDataSize();
    }
  }

View Full Code Here

Examples of com.linkedin.camus.etl.kafka.common.EtlRequest

      requests = new ArrayList<CamusRequest>();
      int size = arg0.readInt();
      for (int i = 0; i < size; i++){
        
        //TODO: factor out kafka specific request functionality 
        CamusRequest request = new EtlRequest();
        request.readFields(arg0);
        requests.add(request);
      }
    }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.