how to download dse.jar

问题:

I am trying to use DataStax Enterprise 4.6 to write a Spark application in Java and run it in DSE's Spark analytics mode.

The code for creating a Spark context using DSEConfHelper:

SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf())
            .setAppName( "My application");

To use DSEConfHelper we need to import com.datastax.bdp.spark.DseSparkConfHelper which is located in dse.jar.

In my pom.xml I have included the dependency:

<dependency>
    <groupId>com.datastax</groupId>
    <artifactId>bdp</artifactId>
    <version>4.6.0</version>
</dependency>

But Maven cannot download dse.jar.

Please help me.

The reference for code for creating a Spark context is taken from: http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/spark/sparkJavaApi.html

回答1:

Edit: This has been entirely superceded by the com.datastax.dse.dse-spark-dependencies artifact. Add it to your pom.xml:

<dependencies>
  <dependency>
    <groupId>com.datastax.dse</groupId>
    <artifactId>dse-spark-dependencies</artifactId>
    <version>${dse.version}</version>
    <scope>provided</scope>
  </dependency>
<dependencies>

<repositories>
  <repository>
    <id>DataStax-Repo</id>
    <url>https://repo.datastax.com/public-repos/</url>
  </repository>
</repositories>

See https://github.com/datastax/SparkBuildExamples for Maven, SBT, and Gradle example projects.

Original, outdated answer:

You have to manually install dse.jar as of right now. There are two ways of doing this.

Option 1 Install the JAR file using mvn install:

$ mvn install:install-file -Dfile=<path-to-dse.jar> -DgroupId=com.datastax -DartficactId=bdp -Dversion=4.6.0

Option 2 Manually copy dse.jar from your install location to ${project.basedir}/lib/. Then modify your pom.xml:

<dependency>
  <groupId>com.datastax</groupId>
  <artifactId>bdp</artifactId>
  <version>4.6.0</version>
  <scope>system</scope>
  <systemPath>${project.basedir}/lib/dse.jar</systemPath>
</dependency>

I don't really know why you're calling the artifact "bdp", but for these purposes it doesn't matter, and I just used it as well.

回答2:

dse.jar is part of DSE installation. If you are working in windows environment, you can find it here dse.jar

register,download and extact to find the jar in lib folder. The use the above answer to add it in your maven project.