From this JIRA ticket Hide UserDefinedType in Spark 2.0
, seems that spark hide the UDT API from version 2.0.
Is there exists an alternative function or API we can use in version 2.2, so that we could define UserDefinedType? I wish to use a custom type in dataframe or structured streaming.
There is no alternative API and UDT remains private (https://issues.apache.org/jira/browse/SPARK-7768).
Generic Encoders (org.apache.spark.sql.Encoders.kryo and org.apache.spark.sql.Encoders.javaSerialization) serve similar purpose in Dataset, but there are not direct replacement:
- How to store custom objects in Dataset?
- Questions about the future of UDTs and Encoders
