How to schedule my Apache Spark application to run

Question

Hi All & IBM Bluemix team, I am using IBM Analytics for Apache Spark service in IBM Bluemix. I have developed a Apache Spark application and I want to run everyday at 00.30 AM in the night. How to schedule my Apache Spark application to run everyday at 00.30 AM(night) in IBM Bluemix?

Answer 1

You can use any scheduling tool like (Crontab on linux) that will allow you to run spark-submit.sh script from your machine at a specific time.(in your case 00:30am)

A typical crontab entry would look like this

30 0 * * * /<your-directory-where-spraksubmit.sh>/spark-submit.sh --vcap vcap.json --deploy-mode cluster --class org.apache.spark.examples.SparkPi --master https://169.54.219.20 spark-examples-1.6.0-hadoop2.6.0.jar 

Refrence to IBM spark-submit.sh https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html

Crontab:-

http://kvz.io/blog/2007/07/29/schedule-tasks-on-linux-using-crontab/

Again you can use any tools of your desire.

Thanks, Charles.