Execute the scala script through spark-shell in si

Question

Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.

I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.

spark-shell -i "file.scala"

after execution, I get into

scala>

I don't want to get into the scala> mode

Updating (October 2019) for a script that terminates

This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script).
See this question with a good example.

It also needs a "silent mode", it is expected to not pollute the output.txt.

Suppose Spark v2.2+.

---

PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.

Answer 1

Adding onto @rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:

spark-shell < yourfile.scala

But this would require you to keep the lines of code within a line in case a statement is written on different lines.

OR

echo :quit | spark-shell -i yourfile.scala

This should

Answer 2

spark-shell -i file.scala keeps the interpreter open in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.

If logging is requiered you can use something like this:

spark-shell < file.scala > test.log 2>&1 &

If you have limitations on editing file and you can't add System.exit(0), use:

echo :quit | scala-shell -i file.scala

UPD

If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.

Answer 3

The best way is definitively to compile your scala code to a jar and use spark-submit but if you're simply looking for a quick iteration loop, you can simply issue a :quit after parsing your scala code:

echo :quit | scala-shell -i yourfile.scala