What's hive-site.xml including in $SPARK_HOME

问题:

I am a beginner at hive, something happened (can not find table) when I start spark job and read data from hive. I don't set hive-site.xml in $SPARK_HOME/conf ?

submit the spark job command is here

bin/spark-submit --master local[*] --driver-memory 8g --executor-memory 8g  --class com.ctrip.ml.client.Client /root/GitLab/di-ml-tool/target/di-ml-tool-1.0-SNAPSHOT.jar

回答1:

On your hive distribution you have a template file that must convert into your own site file.

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-hive-site.xmlandhive-default.xml.template

So first of all you must create your own hive-site.xml file by copying the hive-default.xml.template and then you can use it from spark.

If you don't want to use the default file, you can use any of the previous configurations shown on the previous answers.

回答2:

hive-site.xml look like the following .

<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>USER</value>
        <description>user name for connecting to mysql server </description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>PASSWORD</value>
        <description>password for connecting to mysql server </description>
    </property>
</configuration>

回答3:

Just copy the hive-site.xml to conf directory and it will work.

回答4:

I believe it might depend on the distribution you're using. I encountered this problem recently, and this fixed the issue for me. I'm using HDP 2.3.2 so my copy of hive-site.xml in the Spark conf folder only contains this:

<configuration>

  <property>
  <name>hive.metastore.uris</name>
  <value>thrift://host.xxx.com:9083</value>
  </property>

</configuration>