Scala Object with ArrayBuffer - Deserialize issue

问题:

I have a top level scala class something like below:

FinalOutput.scala:

class FinalOutput extends Serializable {

  @BeanProperty
  var userId: String = _

  @BeanProperty
  var tenantId: String = _

  @BeanProperty
  @SerializedName("type")
  var dataType: String = _

  @BeanProperty
  var data: FinalData = _

  @BeanProperty
  var userCreatedDate: String = _
}

FinalData.scala :

class FinalData extends Serializable {

  @BeanProperty
  var list1: ArrayBuffer[DataType1] = _

  @BeanProperty
  var list2: ArrayBuffer[DataType2] = _

  @BeanProperty
  var list3: ArrayBuffer[DataType3] = _

  @BeanProperty
  var list4: ArrayBuffer[DataType4] = _

  ....
  ....

  @BeanProperty
  var list15: ArrayBuffer[DataType15] = _

  @BeanProperty
  var userName: String = _
}

and all DataType* classes extending BaseBean

I have used this to serialize Scala object into json string.

ArrayBufferSerializer.scala

class ArrayBufferSerializer[T: ClassTag] extends JsonSerializer[ArrayBuffer[T]] {
  override def serialize(src: ArrayBuffer[T], typeOfSrc: Type, context: JsonSerializationContext): JsonElement = {
    context.serialize(src.toArray[Any])
  }
}

then serializing into string using this:

val gson = new GsonBuilder().registerTypeAdapter(classOf[ArrayBuffer[FinalData]], new ArrayBufferSerializer[FinalData]()).serializeNulls.create
val data = gson.toJson(row)

Now I wanted to do the deserialize the json string to FinalOutput object, so I have created ArrayBufferDeSerializer something like this

class ArrayBufferDeSerializer[T: ClassTag] extends JsonDeserializer[ArrayBuffer[T]] {
  override def deserialize(json: JsonElement, typeOfT: Type, context: JsonDeserializationContext): ArrayBuffer[T] = {
    context.deserialize(json, typeOfT)
  }
}

and then calling the below to deserialzie:

val gson = new GsonBuilder().registerTypeAdapter(classOf[ArrayBuffer[FinalData]], new ArrayBufferSerializer[FinalData]()).serializeNulls.create
gson.fromJson(row, classOf[FinalLevelOneSmsOutput])

getting the following error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.StackOverflowError
    at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:720)
    at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:743)
    at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:735)
    at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:718)
    at com.google.gson.internal.Streams.parse(Streams.java:48)
    at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:54)
    at com.google.gson.Gson.fromJson(Gson.java:861)
    at com.google.gson.Gson.fromJson(Gson.java:926)
    at com.google.gson.Gson$1.deserialize(Gson.java:131)
    at com.cv.util.ArrayBufferDeSerializer.deserialize(ArrayBufferDeSerializer.scala:15)
    at com.cv.util.ArrayBufferDeSerializer.deserialize(ArrayBufferDeSerializer.scala:13)
    at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)

回答1:

Your Deserializer does nothing but delegate the deserialization back to the context with the same arguments (same json and same type), which would cause the context to call the deserializer again - which creates the infinite loop and the resulting StackOverflowError.

The deserializer has to be improved - since we've serialized ArrayBuffers into "simple" arrays, we have to deserialize them accordingly. Here's one way to do it:

import com.google.gson.reflect.TypeToken

class ArrayBufferDeSerializer[T: ClassTag] extends JsonDeserializer[ArrayBuffer[T]] {
  override def deserialize(json: JsonElement, typeOfT: Type, context: JsonDeserializationContext): ArrayBuffer[T] = {
    // since we've serialized ArrayBuffers by converting them to simple Arrays, we deserialize the
    // input as a simple Array first:
    val array: util.ArrayList[T] = context.deserialize(json, new TypeToken[Array[T]](){}.getType)

    // Then, we convert it back into an ArrayBuffer:
    import collection.JavaConverters._
    ArrayBuffer[T](array.asScala: _*)
  }
}