Convert RDD into Dataframe in pyspark

问题:

I am trying to convert my RDD into Dataframe in pyspark.

My RDD:

[(['abc', '1,2'], 0), (['def', '4,6,7'], 1)]

I want the RDD in the form of a Dataframe:

Index Name Number
 0    abc   [1,2]
 1    def   [4,6,7]

I tried:

rd2=rd.map(lambda x,y: (y, x[0] , x[1]) ).toDF(["Index", "Name" , "Number"])

But I am getting errors

 An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 62.0 failed 1 times, most recent failure: Lost task 0.0 
in stage 62.0 (TID 88, localhost, executor driver): 
org.apache.spark.api.python.PythonException: Traceback (most recent 
call last):

Can you let me know, where am I going wrong?

Update:

rd2=rd.map(lambda x: (x[1], x[0][0] , x[0][1]))

I have the RDD in the form :

[(0, 'abc', '1,2'), (1, 'def', '4,6,7')]

To convert to Dataframe:

rd2.toDF(["Index", "Name" , "Number"])

Its still giving me error:

An error occurred while calling o2271.showString.
: java.lang.IllegalStateException: SparkContext has been shutdown
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2021)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2050)

回答1:

RDD.map takes an unary function:

rdd.map(lambda x: (x[1], x[0][0] , x[0][1])).toDF(["Index", "Name" , "Number"])

so you cannot pass binary one.

If you want to split array:

rdd.map(lambda x: (x[1], x[0][0] , x[0][1].split(","))).toDF(["Index", "Name" , "Number"])