how to register kryo serializer instances in storm

问题:

I'm desparately trying to configure serializer instances to use in my storm topology.

The storm documentation states, there are 2 ways to register serializers :

1. The name of a class to register. In this case, Storm will use Kryo’s FieldsSerializer to serialize the class. This may or may not be optimal for the class – see the Kryo docs for more details.
2. A map from the name of a class to register to an implementation of com.esotericsoftware.kryo.Serializer.

I want to use 2. ->

Map<String, Object> serializerConfig = new HashMap<String, Object>();
serializerConfig.put(Record.class.getName(), new AvroSerializer(params));
conf.put(Config.TOPOLOGY_KRYO_REGISTER, serializerConfig);

Unfortunately, this results in

Exception in thread "main" java.lang.IllegalArgumentException: Storm conf is not valid. Must be json-serializable

on topology submission.

Does anyone know how to do this (register serializer instances) ?

Thank you very much

回答1:

There is a method in the backtype.storm.Config class to register your own class derived from Serializer.

For your example, put this in the main method that creates and submits the topology:

// Read in Storm Configuration
Config conf = new Config();
conf.registerSerialization(Record.class, AvroSerializer.class);

回答2:

As Steven Magana-Zook said above, you want to register the class in the config as he's done. This apparently doesn't let you pass in parameters, but if you look in SerializationFactory.java in storm's source, you can see that it resolves various possible constructors of your serializer class, including several that contain the storm Config. You can stash your parameters in there.

So not what you were hoping for exactly but you should be able to reach the same end.