Decompressing LZ4 compressed data in Spark

Question

I have LZ4 compressed data in HDFS and I'm trying to decompress it in Apache Spark into a RDD. As far as I can tell, the only method in JavaSparkContext to read data from HDFS is textFile which only reads data as it is in HDFS. I have come across articles on CompressionCodec but all of them explain how to compress output to HDFS whereas I need to decompress what is already on HDFS.

I am new to Spark so I apologize in advance if I missed something obvious or if my conceptual understanding is incorrect but it would be great if someone could point me in the right direction.

Answer 1

Spark 1.1.0 supports reading LZ4 compressed files via sc.textFile. I've got it working by using Spark that is built with Hadoop that supports LZ4 (2.4.1 in my case)

After that, I've built native libraries for my platform as described in Hadoop docs and linked them them to Spark via --driver-library-path option.

Without linking there were native lz4 library not loaded exceptions.

Depending on Hadoop distribution you are using building native libraries step may be optional.