Submit Spark job is not loading spark-cloudant:2.0

问题:

I am submitting Spark using below command,

./spark-submit --packages cloudant-labs:spark-cloudant:2.0.0-s_2.11 --class spark.cloudant.connecter.cloudantconnecter --master local[*] /opt/demo/sparkScripts/ScoredJob/sparkcloudantconnecter.jar

But it seems "spark-cloudant" package is not loading.. Might be it is not loading because of artifact located at Spark Packages repository(https://dl.bintray.com/spark-packages/maven/)

So What changes is require in command to load the package?

Full details of This package is showing here https://mvnrepository.com/artifact/cloudant-labs/spark-cloudant/2.0.0-s_2.11

回答1:

It seems require to add repository while loading package using Spark-submit command,

-- repository url

So new command will be,

./spark-submit --packages cloudant-labs:spark-cloudant:2.0.0-s_2.11 --repositories https://dl.bintray.com/spark-packages/maven/ --class spark.cloudant.connecter.cloudantconnecter --master local[*] /opt/demo/sparkScripts/ScoredJob/sparkcloudantconnecter.jar