Get list of data types from schema in Apache Spark

2019-03-25 09:32发布

I have the following code in Spark-Python to get the list of names from the schema of a DataFrame, which works fine, but how can I get the list of the data types?

columnNames = df.schema.names

For example, something like:

columnTypes = df.schema.types

Is there any way to get a separate list of the data types contained in a DataFrame schema?

3条回答
叼着烟拽天下
2楼-- · 2019-03-25 09:51

Use schema.dtypes

scala> val df = Seq(("ABC",10,20.4)).toDF("a","b","c")
df: org.apache.spark.sql.DataFrame = [a: string, b: int ... 1 more field]

scala>

scala> df.printSchema
root
 |-- a: string (nullable = true)
 |-- b: integer (nullable = false)
 |-- c: double (nullable = false)

scala> df.dtypes
res2: Array[(String, String)] = Array((a,StringType), (b,IntegerType), (c,DoubleType))

scala> df.dtypes.map(_._2).toSet
res3: scala.collection.immutable.Set[String] = Set(StringType, IntegerType, DoubleType)

scala>
查看更多
戒情不戒烟
3楼-- · 2019-03-25 09:56

Since the question title is not python-specific, I'll add scala version here:

val tyes = df.schema.fields.map(f => f.dataType)

It will result in an array of org.apache.spark.sql.types.DataType.

查看更多
啃猪蹄的小仙女
4楼-- · 2019-03-25 10:05

Here's a suggestion:

df = sqlContext.createDataFrame([('a', 1)])

types = [f.dataType for f in df.schema.fields]

types
> [StringType, LongType]

Reference:

查看更多
登录 后发表回答