How to deploy Spark Streaming application?

Question

I need to deploy a Spark Streaming application on Linux server.

Can anyone provide the steps to how to deploy and what code modification require before deploy?

class JavaKafkaWordCount11 {
    public static void main(String[] args) {
        StreamingExamples.setStreamingLogLevels();

        SparkConf sparkConf = new SparkConf()
                .setAppName("JavaKafkaWordCount11")
                .setMaster("local[*]");
        sparkConf.set("spark.streaming.concurrentJobs", "20");

        JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(1500));

        Map<String, Integer> topicMap = new HashMap<>();
        topicMap.put("TopicQueue", 20);
        JavaPairReceiverInputDStream<String, String> messages =
                KafkaUtils.createStream(jssc, "x.xx.xxx.xxx:2181", "1", topicMap);
        JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
            @Override
            public String call(Tuple2<String, String> tuple2) {
                return tuple2._2();
            }
        });
        lines.foreachRDD(rdd -> {
            if (rdd.count() > 0) {
                List<String> strArray = rdd.collect();
                getProcessResult(strArray);
            }
        });
    }
}

Answer 1

Here are the steps:

1. Read Quick Start

Yes, there's only one step required that boils down to:

1. sbt package which assumes you use sbt which for Java might be gradle or maven, though. That just says you have to package your Spark application so it's ready for deployment.

2. spark-submit your packaged Spark application.

You may optionally start your cluster (e.g. Spark Standalone, Apache Mesos or Hadoop YARN), but it's not really needed since spark-submit assumes local[*] by default.

p.s. You're using Apache Kafka so you have to have it up and running (at x.xx.xxx.xxx:2181), too.

Answer 2

You can submit your job through Spark-submit.like this..,

./spark-submit --class packagename.classname [--jars path to any external jars] --master local[4] "Your jar file path"

for any reference follow this link:

Spark-submit

Thanks.