How to add a column to the beginning of the schema

问题:

Dataset.withColumn() seems to append the column to the end of the schema. How to prepend the column to the beginning of the schema?

回答1:

Use select with wildcard:

df.select(new_column, col("*"))

回答2:

Above answer is right but one catch is duplicate column name is coming at the first and last... so tried with another variation, using selectExpr (Selects a set of SQL expressions is)

    val someData = Seq(
      Row(8, "Ram"),
      Row(64, "erwaman"),
      Row(-27, "user8371915")
    )

    val someSchema = List(
      StructField("number", IntegerType, true),
      StructField("word", StringType, true)
    )

    val someDF = spark.createDataFrame(
      spark.sparkContext.parallelize(someData),
      StructType(someSchema)
    )
    println("initial")
    someDF.show(false)


    println("mycolumn added at the end")
    val dataframecolumnOrderChanged = someDF.withColumn("mycolumn", functions.lit("mycolumnadded"))
    dataframecolumnOrderChanged.show(false)
    println("mycolumn added at the beginning")
    moveColumnToFirstPos(dataframecolumnOrderChanged, "mycolumn")


 /**
    * moveColumnToFirstPos : This function can be used after withColumn applied to a data frame
    * which will add column to the right most position
    * if you want to move column to first position in the dataframe selection and drop duplicate as well
    *
    * @param dataframecolumnOrderChanged
    */
   def moveColumnToFirstPos(dataframecolumnOrderChanged: DataFrame, colname: String) = {
    val xs = Array(colname) ++ dataframecolumnOrderChanged.columns.dropRight(1)
    val fistposColDF = dataframecolumnOrderChanged.selectExpr(xs: _*)
    fistposColDF.show(false)
    fistposColDF
  }

---

Result :

initial
+------+-----------+
|number|word       |
+------+-----------+
|8     |Ram        |
|64    |erwaman    |
|-27   |user8371915|
+------+-----------+

mycolumn added at the end
+------+-----------+-------------+
|number|word       |mycolumn     |
+------+-----------+-------------+
|8     |Ram        |mycolumnadded|
|64    |erwaman    |mycolumnadded|
|-27   |user8371915|mycolumnadded|
+------+-----------+-------------+

mycolumn added at the beginning
+-------------+------+-----------+
|mycolumn     |number|word       |
+-------------+------+-----------+
|mycolumnadded|8     |Ram        |
|mycolumnadded|64    |erwaman    |
|mycolumnadded|-27   |user8371915|
+-------------+------+-----------+