Convert between spark.SQL DataFrame and pandas Dat

问题:

This question already has an answer here:

• Requirements for converting Spark dataframe to Pandas/R dataframe 1 answer

Is that possible to

convert from to pd.DataFrame

under %pyspark environment ?

回答1:

Try:

spark_df.toPandas()

toPandas()

Returns the contents of this DataFrame as Pandas pandas.DataFrame.

This is only available if Pandas is installed and available.

And if you want the oposite:

spark_df = createDataFrame(pandas_df)