I have a DataFrame
with many columns of str
type, and I want to apply a function to all those columns, without renaming their names or adding more columns, I tried using a for-in
loop executing withColumn
(see example bellow), but normally when I run the code, it shows a Stack Overflow
(it rarely works), this DataFrame
is not big at all, it has just ~15000 records.
# df is a DataFrame
def lowerCase(string):
return string.strip().lower()
lowerCaseUDF = udf(lowerCase, StringType())
for (columnName, kind) in df.dtypes:
if(kind == "string"):
df = df.withColumn(columnName, lowerCaseUDF(df[columnName]))
df.select("Tipo_unidad").distinct().show()
The complete error is very long, therefore I decided to paste only some lines. But you can find the full trace here Complete Trace
Py4JJavaError: An error occurred while calling o516.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2.0 failed 4 times, most recent failure: Lost task 1.3 in stage 2.0 (TID 38, worker2.mcbo.mood.com.ve): java.lang.StackOverflowError at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2774)
I am thinking that this problem is produced because this code launches many jobs (one for each column of type string
), could you show me another alternative or what I am doing wrong?
Try something like this:
This approach has two main advantages over you current solution:
BatchPythonProcessing
).