Append/Prepend string to a large file in Scala

2019-05-30 01:14发布

I've got a large file that after some spark processing has a header inside it(but not at the top) so I have to append this string header to the first line of the file.

I tried to do this since I could not find any way to prepend directly to first line (first I read all large content, then I clear the entire file and finally overwrite the file with the header and then append all the large string)

val content: String = txtOutput.contentAsString
    txtOutput.clear()
  txtOutput.overwrite(model_header)
  txtOutput.appendLine().append(content)

But I get the following problem "Requested array size exceeds VM limit" that I can't manage since I am coding in Databricks so I cannot tweak it, so I need any way to process my data

0条回答
登录 后发表回答