Protocol buffer objects generated at runtime

Question

a colleague of mine came with an idea of generating protocol buffers classes at runtime. Meaning:

• There is C++ server application and Java client application communicating over TCP/IP via protocol buffers messages.
• The C++ application may have different schema in different versions and this is not necessarily backward compatible
• There is Java application communicating with this server which should support all possible server versions.

The idea is that the server sends the protocol buffer's definition as part of the initial handshake and the java application generates the class at runtime and use it for communication with the server.

I wonder whether this is even vital idea and if there is possibly some utility for such use case.

Thanks

Answer 1

What you describe is actually already supported by the Protocol Buffers implementations in C++ and Java. All you have to do is transmit a FileDescriptorSet (as defined in google/protobuf/descriptor.proto) containing the FileDescriptorProtos representing each relevant .proto file, then use DynamicMessage to interpret the messages on the receiving end.

To get a FileDescriptorProto in C++, given message type Foo that is defined in that file, do:

google::protobuf::FileDescriptorProto file;
Foo::descriptor().file()->CopyTo(&file);

Put all the FileDescriptorProtos that define the types you need, plus all the files that they import, into a FileDescriptorSet proto. Note that you can use google::protobuf::FileDescriptor (the thing returned by Foo::descriptor().file()) to iterate over dependencies rather than explicitly name each one.

Now, send the FileDescriptorSet to the client.

On the client, use FileDescriptor.buildFrom() to convert each FileDescriptorProto to a live Descriptors.FileDescriptor. You will have to make sure to build dependencies before dependents, since you have to provide the already-built dependencies to buildFrom() when building the dependents.

From there, you can use the FileDescriptor's findMessageTypeByName() to find the Descriptor for the specific message type you care about.

Finally, you can call DynamicMessage.newBuilder(descriptor) to construct a new builder instance for the type in question. DynamicMessage.Builder implements the Message.Builder interface, which has fields like getField() and setField() to manipulate the fields of the message dynamically (by specifying the corresponding FieldDescriptors).

Similarly, you can call DynamicMessage.parseFrom(descriptor,input) to parse messages received from the server.

Note that one disadvantage of DynamicMessage is that it is relatively slow. Essentially, it's like an interpreted language. Generated code is faster because the compiler can optimize for the specific type, whereas DynamicMessage has to be able to handle any type.

However, there's really no way around this. Even if you ran the code generator and compiled the class at runtime, the code which actually uses the new class would still be code that you wrote earlier, before you knew what type you were going to use. Therefore, it still has to use a reflection or reflection-like interface to access the message, and that is going to be slower than if the code were hand-written for the specific type.

But is it a good idea?

Well, this depends. What is the client actually going to do with this schema it receives from the server? Transmitting a schema over the wire doesn't magically make the client compatible with that version of the protocol -- the client still has to understand what the protocol means. If the protocol has been changed in a backwards-incompatible way, this almost certainly means that the meaning of the protocol has changed, and the client code has to be updated, schema transmission or not. The only time where you can expect the client to continue working without an update is when the client is only doing a generic operation that only depends on the message content but not the message meaning -- for example, the client could convert the message to JSON without having to know what it means. But this is relatively unusual, particularly on the client end of an application. This is exactly why Protobufs doesn't send any type information by default -- because it's usually useless, since if the receiver doesn't know the meaning, the schema is irrelevant.

If the issue is that the server is sending messages to the client which aren't intended to be interpreted at all, but just sent back to the server at a later time, then the client doesn't need the schema at all. Just transmit the message as bytes and don't bother parsing it. Note that a bytes field containing an encoded message of type Foo looks exactly the same on the wire as a field whose type is actually declared as Foo. You could actually compile the client and server against slightly different versions of the .proto file, where the client sees a particular field as bytes while the server sees it as a sub-message, in order to avoid the need for the client to be aware of the definition of that sub-message. ``

Answer 2

For Java you may find following wrapper API ("protobuf-dynamic") easier to use than the original protobuf API:

https://github.com/os72/protobuf-dynamic

For example:

// Create dynamic schema
DynamicSchema.Builder schemaBuilder = DynamicSchema.newBuilder();
schemaBuilder.setName("PersonSchemaDynamic.proto");

MessageDefinition msgDef = MessageDefinition.newBuilder("Person") // message Person
    .addField("required", "int32", "id", 1)     // required int32 id = 1
    .addField("required", "string", "name", 2)  // required string name = 2
    .addField("optional", "string", "email", 3) // optional string email = 3
    .build();

schemaBuilder.addMessageDefinition(msgDef);
DynamicSchema schema = schemaBuilder.build();

// Create dynamic message from schema
DynamicMessage.Builder msgBuilder = schema.newMessageBuilder("Person");
Descriptor msgDesc = msgBuilder.getDescriptorForType();
DynamicMessage msg = msgBuilder
    .setField(msgDesc.findFieldByName("id"), 1)
    .setField(msgDesc.findFieldByName("name"), "Alan Turing")
    .setField(msgDesc.findFieldByName("email"), "[e-mail]")
    .build();

Dynamic schemas can be useful in some applications to distribute changes without recompiling code (say in a more dynamically typed system). They can also be very useful for "dumb" applications that require no semantic understanding (say a data browser tool)