Examples of AvroDatumConverterFactory

org.apache.avro.hadoop.io.AvroDatumConverterFactory
Constructs converters that turn objects (usually from the output of a MR job) into Avro data that can be serialized.
Currently, only the following types have implemented converters:
- AvroKey
- AvroValue
- BooleanWritable
- BytesWritable
- ByteWritable
- DoubleWritable
- FloatWritable
- IntWritable
- LongWritable
- NullWritable
- Text

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

    AvroJob.setOutputValueSchema(job, TextStats.SCHEMA$);
    TaskAttemptContext context = createMock(TaskAttemptContext.class);


    replay(context);


    AvroDatumConverterFactory factory = new AvroDatumConverterFactory(job.getConfiguration());
    AvroDatumConverter<Text, ?> keyConverter = factory.create(Text.class);
    AvroValue<TextStats> avroValue = new AvroValue<TextStats>(null);
    @SuppressWarnings("unchecked")
    AvroDatumConverter<AvroValue<TextStats>, ?> valueConverter
        = factory.create((Class<AvroValue<TextStats>>) avroValue.getClass());
    CodecFactory compressionCodec = CodecFactory.nullCodec();
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();


    // Use a writer to generate a Avro container file in memory.
    // Write two records: <'apple', TextStats('apple')> and <'banana', TextStats('banana')>.

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

    R1 record = new R1();
    record.attribute = "test";
    AvroValue<R1> avroValue = new AvroValue<R1>(record);


    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    AvroDatumConverterFactory factory =
      new AvroDatumConverterFactory(job.getConfiguration());


    AvroDatumConverter<Text, ?> keyConverter = factory.create(Text.class);


    @SuppressWarnings("unchecked")
    AvroDatumConverter<AvroValue<R1>, R1> valueConverter =
      factory.create((Class<AvroValue<R1>>) avroValue.getClass());


    AvroKeyValueRecordWriter<Text, AvroValue<R1>> writer =
      new AvroKeyValueRecordWriter<Text, AvroValue<R1>>(
        keyConverter, valueConverter, new ReflectData(),
        CodecFactory.nullCodec(), outputStream);

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

  private static final Log LOG = LogFactory.getLog(SequenceFileSerDe.class);
  AvroDatumConverter valADC = null;


  void initDeserializer(String valClassNamePayload) {
    // none necessary
    AvroDatumConverterFactory adcFactory = new AvroDatumConverterFactory(new Configuration());
    try {
      this.valADC = adcFactory.create(Class.forName(valClassNamePayload));
    } catch (ClassNotFoundException cnfe) {
      cnfe.printStackTrace();
    }
  }

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory


  /**
   * Figure out the schema for non-Avro SequenceFiles.  Yields a synthesized Avro schema.
   */
  void computeSchema() throws IOException {
    AvroDatumConverterFactory adcFactory = new AvroDatumConverterFactory(new Configuration());
    try {
      SequenceFile.Reader in = new SequenceFile.Reader(FSAnalyzer.getInstance().getFS(), dd.getFilename(), new Configuration());
      try {
        //
        // Build Avro schemas out of the SequenceFile key/val classes.  We will use
        // these them to display schema information to the user.
        //
        Class keyClass = in.getKeyClass();
        Class valClass = in.getValueClass();
        this.valClassName = valClass.getName();


        AvroDatumConverter keyADC = adcFactory.create(keyClass);
        AvroDatumConverter valADC = adcFactory.create(valClass);
        
        //
        // Build a "pair record" with "key" and "value" fields to hold the subschemas.
        //
        List<Schema.Field> fieldList = new ArrayList<Schema.Field>();

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

      Class keyClass;
      Class valClass;
      AvroDatumConverter keyADC;
      AvroDatumConverter valADC;      
      {
        AvroDatumConverterFactory adcFactory = new AvroDatumConverterFactory(new Configuration());
        try {
          in = new SequenceFile.Reader(FSAnalyzer.getInstance().getFS(), dd.getFilename(), new Configuration());
          keyClass = in.getKeyClass();
          valClass = in.getValueClass();
          keyADC = adcFactory.create(keyClass);
          valADC = adcFactory.create(valClass);
          
          nextElt = lookahead();
        } catch (IOException iex) {
          this.nextElt = null;
        }

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

  
  /** {@inheritDoc} */
  @SuppressWarnings("unchecked")
  @Override
  protected Schema initSchema(TaskAttemptContext context) {
    AvroDatumConverterFactory converterFactory = new AvroDatumConverterFactory(
        context.getConfiguration());
    
    keyConverter = converterFactory.create((Class<K>) context
        .getOutputKeyClass());
    valueConverter = converterFactory.create((Class<V>) context
        .getOutputValueClass());


    // Create the generic record schema for the key/value pair.
    return AvroKeyValue.getSchema(
        keyConverter.getWriterSchema(), valueConverter.getWriterSchema());

View Full Code Here

Examples of org.apache.avro.hadoop.io.AvroDatumConverterFactory

  @Override
  @SuppressWarnings("unchecked")
  public RecordWriter<K, V> getRecordWriter(TaskAttemptContext context) throws IOException {
    Configuration conf = context.getConfiguration();


    AvroDatumConverterFactory converterFactory = new AvroDatumConverterFactory(conf);


    AvroDatumConverter<K, ?> keyConverter = converterFactory.create(
        (Class<K>) context.getOutputKeyClass());
    AvroDatumConverter<V, ?> valueConverter = converterFactory.create(
        (Class<V>) context.getOutputValueClass());


    GenericData dataModel = AvroSerialization.createDataModel(conf);


    return new AvroKeyValueRecordWriter<K, V>(keyConverter, valueConverter,

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.