如何创建地图的数据集呢？如何创建地图的数据集呢？(How to create a Dataset o

我使用的Spark 2.2，并试图打电话的时候，我遇到了麻烦spark.createDataset对Seq的Map 。

代码和输出从我的星火壳牌会话如下：

// createDataSet on Seq[T] where T = Int works
scala> spark.createDataset(Seq(1, 2, 3)).collect
res0: Array[Int] = Array(1, 2, 3)

scala> spark.createDataset(Seq(Map(1 -> 2))).collect
<console>:24: error: Unable to find encoder for type stored in a Dataset.  
Primitive types (Int, String, etc) and Product types (case classes) are 
supported by importing spark.implicits._
Support for serializing other types will be added in future releases.
       spark.createDataset(Seq(Map(1 -> 2))).collect
                          ^

// createDataSet on a custom case class containing Map works
scala> case class MapHolder(m: Map[Int, Int])
defined class MapHolder

scala> spark.createDataset(Seq(MapHolder(Map(1 -> 2)))).collect
res2: Array[MapHolder] = Array(MapHolder(Map(1 -> 2)))

我已经试过import spark.implicits._ ，虽然我相当肯定，含蓄真实由星火shell会话进口。

这是没有覆盖当前的编码器的情况下？

Answer 1:

这是不包括在2.2，但可以很容易地解决。 您可以添加所需的Encoder使用ExpressionEncoder ，无论是明确：

import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder  
import org.apache.spark.sql.Encoder

spark
  .createDataset(Seq(Map(1 -> 2)))(ExpressionEncoder(): Encoder[Map[Int, Int]])

或implicitly ：

implicit def mapIntIntEncoder: Encoder[Map[Int, Int]] = ExpressionEncoder()
spark.createDataset(Seq(Map(1 -> 2)))

Answer 2:

仅供参考，上述表达只是工作在星火2.3（截至本承诺 ，如果我没有记错）。

scala> spark.version
res0: String = 2.3.0

scala> spark.createDataset(Seq(Map(1 -> 2))).collect
res1: Array[scala.collection.immutable.Map[Int,Int]] = Array(Map(1 -> 2))

我想这是因为newMapEncoder现在的一部分spark.implicits 。

scala> :implicits
...
  implicit def newMapEncoder[T <: scala.collection.Map[_, _]](implicit evidence$3: reflect.runtime.universe.TypeTag[T]): org.apache.spark.sql.Encoder[T]

通过下面的技巧你可以“禁用”隐，并给予上述表达式一试（这将导致一个错误）。

trait ThatWasABadIdea
implicit def newMapEncoder(ack: ThatWasABadIdea) = ack

scala> spark.createDataset(Seq(Map(1 -> 2))).collect
<console>:26: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
       spark.createDataset(Seq(Map(1 -> 2))).collect
                          ^

文章来源: How to create a Dataset of Maps?