Java: JSON -> Protobuf & back conversion

Question

I have an existing system, which is using protobuf-based communication protocol between GUI and server. Now I would like to add some persistence, but at the moment protobuf messages are straight converted to a third-party custom objects.

Is there a way to convert proto messages to json, which could be then persisted to database.

N.B.: I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

Answer 1

Generics Solution

Here's a generic version of Json converter

package com.github.platform.util;

import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import com.google.protobuf.AbstractMessage.Builder;
import com.google.protobuf.Message;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;

/**
 * Generic ProtoJsonUtil to be used to serialize and deserialize Proto to json
 * 
 * @author [e-mail]
 *
 */
public final class ProtoJsonUtil {

  /**
   * Makes a Json from a given message or builder
   * 
   * @param messageOrBuilder is the instance
   * @return The string representation
   * @throws IOException if any error occurs
   */
  public static String toJson(MessageOrBuilder messageOrBuilder) throws IOException {
    return JsonFormat.printer().print(messageOrBuilder);
  }

  /**
   * Makes a new instance of message based on the json and the class
   * @param <T> is the class type
   * @param json is the json instance
   * @param clazz is the class instance
   * @return An instance of T based on the json values
   * @throws IOException if any error occurs
   */
  @SuppressWarnings({"unchecked", "rawtypes"})
  public static <T extends Message> T fromJson(String json, Class<T> clazz) throws IOException {
    // https://stackoverflow.com/questions/27642021/calling-parsefrom-method-for-generic-protobuffer-class-in-java/33701202#33701202
    Builder builder = null;
    try {
      // Since we are dealing with a Message type, we can call newBuilder()
      builder = (Builder) clazz.getMethod("newBuilder").invoke(null);

    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException
        | NoSuchMethodException | SecurityException e) {
      return null;
    }

    // The instance is placed into the builder values
    JsonFormat.parser().ignoringUnknownFields().merge(json, builder);

    // the instance will be from the build
    return (T) builder.build();
  }
}

Using it is as simple as follows:

Message instance

GetAllGreetings.Builder allGreetingsBuilder = GetAllGreetings.newBuilder();

allGreetingsBuilder.addGreeting(makeNewGreeting("Marcello", "Hi %s, how are you", Language.EN))
        .addGreeting(makeNewGreeting("John", "Today is hot, %s, get some ice", Language.ES))
        .addGreeting(makeNewGreeting("Mary", "%s, summer is here! Let's go surfing!", Language.PT));

GetAllGreetings allGreetings = allGreetingsBuilder.build();

To Json Generic

String json = ProtoJsonUtil.toJson(allGreetingsLoaded);
log.info("Json format: " + json);

From Json Generic

GetAllGreetings parsed = ProtoJsonUtil.fromJson(json, GetAllGreetings.class);
log.info("The Proto deserialized from Json " + parsed);

Answer 2

Adding to Ophirs answer, JsonFormat is available even before protobuf 3.0. However, the way to do it differs a little bit.

In Protobuf 3.0+, the JsonFormat class is a singleton and therefore do something like the below

String jsonString = "";
JsonFormat.parser().ignoringUnknownFields().merge(json,yourObjectBuilder);

In Protobuf 2.5+, the below should work

String jsonString = "";
JsonFormat jsonFormat = new JsonFormat();
jsonString = jsonFormat.printToString(yourProtobufMessage);

Here is a link to a tutorial I wrote that uses the JsonFormat class in a TypeAdapter that can be registered to a GsonBuilder object. You can then use Gson's toJson and fromJson methods to convert the proto data to Java and back.

Replying to jean . If we have the protobuf data in a file and want to parse it into a protobuf message object, use the merge method TextFormat class. See the below snippet:

// Let your proto text data be in a file MessageDataAsProto.prototxt
// Read it into string  
String protoDataAsString = FileUtils.readFileToString(new File("MessageDataAsProto.prototxt"));

// Create an object of the message builder
MyMessage.Builder myMsgBuilder = MyMessage.newBuilder();

// Use text format to parse the data into the message builder
TextFormat.merge(protoDataAsString, ExtensionRegistry.getEmptyRegistry(), myMsgBuilder);

// Build the message and return
return myMsgBuilder.build();

Answer 3

Try JsonFormat.printer().print(MessageOrBuilder), it looks good for proto3. Yet, it is unclear how to convert the actual protobuf message (which is provided as the java package of my choice defined in the .proto file) to a com.google.protbuf.Message object.

Answer 4

As mentioned in an answer to a similar question, since v3.1.0 this is a supported feature of ProtocolBuffers. For Java, include the extension module com.google.protobuf:protobuf-java-util and use JsonFormat like so:

JsonFormat.parser().ignoringUnknownFields().merge(json, yourObjectBuilder);
YourObject value = yourObjectBuilder.build();

Answer 5

We are currently using protobuf-java-format to convert our Protobuf messages (anything subclass of Message) into a JSON format to send over our web API.

Simply do:

  JsonFormat.printToString(protoMessage)

Answer 6

I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

Converting protobuf to JSON for storage and then back to protobuf on load is much more likely to create compatibility problems, because:

• If the process which performs the conversion is not built with the latest version of the protobuf schema, then converting will silently drop any fields that the process doesn't know about. This is true both of the storing and loading ends.
• Even with the most recent schema available, JSON <-> Protobuf conversion may be lossy in the presence of imprecise floating-point values and similar corner cases.
• Protobufs actually have (slightly) stronger backwards-compatibility guarantees than JSON. Like with JSON, if you add a new field, old clients will ignore it. Unlike with JSON, Protobufs allow declaring a default value, which can make it somewhat easier for new clients to deal with old data that is otherwise missing the field. This is only a slight advantage, but otherwise Protobuf and JSON have equivalent backwards-compatibility properties, therefore you are not gaining any backwards-compatibility advantages from storing in JSON.

With all that said, there are many libraries out there for converting protobufs to JSON, usually built on the Protobuf reflection interface (not to be confused with the Java reflection interface; Protobuf reflection is offered by the com.google.protobuf.Message interface).