What's hive-site.xml including in $SPARK_HOME

Question

I am a beginner at hive, something happened (can not find table) when I start spark job and read data from hive. I don't set hive-site.xml in $SPARK_HOME/conf ?

submit the spark job command is here

bin/spark-submit --master local[*] --driver-memory 8g --executor-memory 8g  --class com.ctrip.ml.client.Client /root/GitLab/di-ml-tool/target/di-ml-tool-1.0-SNAPSHOT.jar

Answer 1

Just copy the hive-site.xml to conf directory and it will work.

Answer 2

hive-site.xml look like the following .

<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>USER</value>
        <description>user name for connecting to mysql server </description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>PASSWORD</value>
        <description>password for connecting to mysql server </description>
    </property>
</configuration>

Answer 3

I believe it might depend on the distribution you're using. I encountered this problem recently, and this fixed the issue for me. I'm using HDP 2.3.2 so my copy of hive-site.xml in the Spark conf folder only contains this:

<configuration>

  <property>
  <name>hive.metastore.uris</name>
  <value>thrift://host.xxx.com:9083</value>
  </property>

</configuration>

Answer 4

On your hive distribution you have a template file that must convert into your own site file.

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-hive-site.xmlandhive-default.xml.template

So first of all you must create your own hive-site.xml file by copying the hive-default.xml.template and then you can use it from spark.

If you don't want to use the default file, you can use any of the previous configurations shown on the previous answers.