Polymorphism and inheritance in Avro schemas

Question

Is it possible to write an Avro schema/IDL that will generate a Java class that either extends a base class or implements an interface? It seems like the generated Java class extends the org.apache.avro.specific.SpecificRecordBase. So, the implements might be the way to go. But, I don't know if this is possible.

I have seen examples with suggestions to define an explicit "type" field in each specific schema, with more of an association than inheritance semantics.

I use my base class heavily in my factory classes and other parts of the code with generics like <T extends BaseObject>. Currently, I had it code generated from the JSON Schema, which supports inheritance.

Another side question: can you use IDL to define just records without the protocol definition? I think the answer is no because the compiler complains about the missing protocol keyword.

Help appreciated! Thanks.

Answer 1

I decided to use the ReflectData API to generate the Schema from the class at runtime and then use the ReflectDatumWriter for serialization. Use of reflection will be slower. But, it looks like the schema is cached internally. I will report back if I see performance issues.

Schema schema = ReflectData.AllowNull.get().getSchema(sourceObject.getClass());
ReflectDatumWriter<T> reflectDatumWriter = new ReflectDatumWriter<>(schema);

DataFileWriter<T> writer = new DataFileWriter<>(reflectDatumWriter);
try {
    writer.setCodec(CodecFactory.snappyCodec());
    writer.create(schema, new File("data.avro"));
    writer.append(sourceObject);
    writer.close();
}
catch (IOException e) {
    // log exception
}

Answer 2

I found this question having similar problem. In my case I needed just to impose marker interface and only to some types (to distinguish particular classes later). Thanks to your answer, I dug deeper into structure of record.vm template. I found out it's possible to define "javaAnnotation": "my.full.AnnotationName" key in .avsc definition JSON. @my.full.AnnotationName is then added to generated class.

Admittedly, this solution is not built on marker interface finally, though for my purpose is good enough and keeping template untouched is big advantage.

Answer 3

I followed https://www.infoq.com/articles/ApacheAvro/ for implementing inheritance. This put lights on polymorphism as well(which i needed).

One point. While declaring {"name": "user", "type": com.navteq.avro.FacebookUser }, make sure you double quote , like {"name": "user", "type": "com.navteq.avro.FacebookUser" },

If i do not do that, then i was getting error like below

> org.apache.avro.SchemaParseException: org.codehaus.jackson.JsonParseException: Unexpected character ('c' (code 99)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

Answer 4

I found a better way to solve this problem. Looking at the Schema generation source in Avro, I figured out that internally the class generation logic uses Velocity schemas to generate the classes.

I modified the record.vm template to also implement my specific interface. There is a way to specify the location of velocity directory using the templateDirectory configuration in the maven build plugin.

I also switched to using SpecificDatumWriter instead of reflectDatumWriter.

<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
   <version>${avro.version}</version>
   <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>schema</goal>
      </goals>
      <configuration>
         <sourceDirectory>${basedir}/src/main/resources/avro/schema</sourceDirectory>
         <outputDirectory>${basedir}/target/java-gen</outputDirectory>
         <fieldVisibility>private</fieldVisibility>
         <stringType>String</stringType>
         <templateDirectory>${basedir}/src/main/resources/avro/velocity-templates/</templateDirectory>
       </configuration>
    </execution>
  </executions>
</plugin>