save each element of rdd in text file hdfs

问题:

I am using spark application. In each element of rdd contains good amount of data. I want to save each element of rdd into multiple hdfs files respectively. I tried rdd.saveAsTextFile("foo.txt") But I will create a single file for whole rdd. rdd size is 10. I want 10 files in hdfs. How can I achieve this??

回答1:

If I understand your question, you can create a custom output format like this

class RDDMultipleTextOutputFormat extends MultipleTextOutputFormat[Any, Any] {
       override def generateActualKey(key: Any, value: Any): Any = NullWritable.get()
       override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String = key.asInstanceOf[String]
}

Then convert your RDD into a key/val one where the key is the file path, and you can use saveAsHadoopFile function insted of saveAsTextFile, like this:

myRDD.saveAsHadoopFile(OUTPUT_PATH, classOf[String], classOf[String],classOf[RDDMultipleTextOutputFormat])