I have an application in SparkSQL which returns large number of rows that are very difficult to fit in memory so I will not be able to use collect function on DataFrame, is there a way using which I can get all this rows as an Iterable instaed of the entire rows as list.
Note: I am executing this SparkSQL application using yarn-client
Generally speaking transferring all the data to the driver looks a pretty bad idea and most of the time there is a better solution out there but if you really want to go with this you can use
toLocalIterator
method on a RDD:Actually you can just use:
df.toLocalIterator
, here is the reference in Spark source code: