窗口功能partitionBy一个列表(Window Functions partitionBy o

我有一个数据帧tableDS在Scala中我能够删除了使用下列主键重复 -

import org.apache.spark.sql.expressions.Window.partitionBy
import org.apache.spark.sql.functions.row_number

val window = partitionBy(primaryKeySeq.map(k => tableDS(k)): _*).orderBy(tableDS(mergeCol).desc)
tableDS.withColumn("rn", row_number.over(window)).where($"rn" === 1).drop("rn")

我需要用Python语言编写了类似的事情。 primaryKeySeq是Python的list。 我想这样的第一条语句 -

from pyspark.sql.window import Window
import pyspark.sql.functions as func

window = Window.partitionBy(primaryKeySeq).orderBy(tableDS[bdtVersionColumnName].desc())
tableDS1=tableDS.withColumn("rn",rank().over(window))

这不会给我正确的结果。

Answer 1:

它得到了解决 - 这是最终的转化。

from pyspark.sql.window import Window
from pyspark.sql.functions import row_number

window = Window.partitionBy(primaryKeySeq).orderBy(tableDS[bdtVersionColumnName].desc())
tableDS1=tableDS.withColumn("rn", row_number.over(window)).where(tableDS["rn"] == 1).drop("rn")

文章来源: Window Functions partitionBy over a list