I've got a large file that after some spark processing has a header inside it(but not at the top) so I have to append this string header to the first line of the file.
I tried to do this since I could not find any way to prepend directly to first line (first I read all large content, then I clear the entire file and finally overwrite the file with the header and then append all the large string)
val content: String = txtOutput.contentAsString
txtOutput.clear()
txtOutput.overwrite(model_header)
txtOutput.appendLine().append(content)
But I get the following problem "Requested array size exceeds VM limit" that I can't manage since I am coding in Databricks so I cannot tweak it, so I need any way to process my data