Scala code doesnt fetch s3 file

问题:

I am trying to run an EMR scalding job and the Scala code is suppose to fetch the content of a text file located in an S3 bucket. The scala.io.source library is messing up with the correct location of the S3 path.

I am giving the parameter runidfile to the EMR job :

--runidfile s3://my-bucket/input.txt

The scala code does the following :

val runid_path = args("runidfile")
val runid = Source.fromFile(runid_path).getLines().mkString

The code somehow doesn't accept the "//" in the S3 path and I get an error:

Caused by: java.io.FileNotFoundException: s3:/my-bucket/input.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at scala.io.Source$.fromFile(Source.scala:90)
at scala.io.Source$.fromFile(Source.scala:75)
at scala.io.Source$.fromFile(Source.scala:53)
at com.move.scalding.userEvents.RecommenderValidator.(RecommenderValidator.scala:37)

Is there any solution or a workaround to this? I tried using Source.fromURL, but S3 is not a valid protocol so it doesn't accept it.

回答1:

The scala.io.Source library is not meant to access files directly from Amazon S3. You need another library for that.

You can use the offical Amazon S3 Java Library. Here is some sample code (copied together from this question and its answers)

val credentials = new BasicAWSCredentials("myKey", "mySecretKey")
val s3Client = new AmazonS3Client(credentials)
val s3Object = s3Client.getObject(new GetObjectRequest("my-bucket", "input.txt"))
val myData = Source.fromInputStream(s3Object.getObjectContent())

val runid = myData.getLines().mkString