I want to do something like this:
df
.withColumn("newCol", <some formula>)
.filter(s"""newCol > ${(math.min(max("newCol").asInstanceOf[Double],10))}""")
Exception I'm getting:
org.apache.spark.sql.Column cannot be cast to java.lang.Double
Can you please suggest me a way to achieve what i want?
I assume
newCol
is already present indf
, then:Instead of
max($"newCol").over(Window.partitionBy())
you can also jjst writemax($"newCol").over()
I think dataframe describe function is what you are looking for.
I'd separate both steps and either:
or