火花选择与别名添加列(spark select and add columns with alias

2019-10-29 18:59发布

我想选择几列,加几列或鸿沟,一些列的空间填充,并将它们存储以新的名称作为别名。 例如,在SQL应该是这样的:

select "   " as col1, b as b1, c+d as e from table 

我怎样才能在星火实现这一目标?

Answer 1:

您也可以使用本机DF功能以及。 例如,给定:

import org.apache.spark.sql.functions._
val df1 = Seq(
 ("A",1,5,3),
 ("B",3,4,2),
 ("C",4,6,3),
 ("D",5,9,1)).toDF("a","b","c","d")

选择列如下:

df1.select(lit(" ").as("col1"),
           col("b").as("b1"),
           (col("c") + col("d")).as("e"))

为您提供了预期的结果:

+----+---+---+
|col1| b1|  e|
+----+---+---+
|    |  1|  8|
|    |  3|  6|
|    |  4|  9|
|    |  5| 10|
+----+---+---+


Answer 2:

在Spark-SQL,你可以做同样的方式。

import org.apache.spark.sql.functions._
val df1 = Seq(
 ("A",1,5,3),
 ("B",3,4,2),
 ("C",4,6,3),
 ("D",5,9,1)).toDF("a","b","c","d")

df1.createOrReplaceTempView("table")
df1.show()

val df2 = spark.sql("select ' ' as col1, b as b1, c+d as e from table ").show()

输入:

    +---+---+---+---+
    |  a|  b|  c|  d|
    +---+---+---+---+
    |  A|  1|  5|  3|
    |  B|  3|  4|  2|
    |  C|  4|  6|  3|
    |  D|  5|  9|  1|
    +---+---+---+---+

输出:

+----+---+---+
|col1| b1|  e|
+----+---+---+
|    |  1|  8|
|    |  3|  6|
|    |  4|  9|
|    |  5| 10|
+----+---+---+


文章来源: spark select and add columns with alias