message:Hive Schema version 1.2.0 does not match m

问题:

enviroment: spark2.11 hive2.2 hadoop2.8.2

hive shell run successfully! and hava no error or warning. but when run application.sh, start failed

/usr/local/spark/bin/spark-submit \
--class cn.spark.sql.Demo \
--num-executors 3 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 3 \
--files /usr/local/hive/conf/hive-site.xml \
--driver-class-path /usr/local/hive/lib/mysql-connector-java.jar \
/usr/local/java/sql/sparkstudyjava.jar \

and the error tips:

Exception in thread "main" java.lang.IllegalArgumentException: Error while 
instantiating 'org.apache.spark.sql.hive.HiveSessionState':
...
Caused by: java.lang.IllegalArgumentException: Error while instantiating 
'org.apache.spark.sql.hive.HiveExternalCatalog':
...
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to 
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: MetaException(message:Hive Schema version 1.2.0 does not match 
metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
...

i try many method to solving this errors, but errors still occurs. how to fix?

回答1:

Probably hive is referring to another version of hive (which is configured differently). Execute below command & see if output is different from /usr/local/hive.

$which hive

If both are same hive directories, add below properties in hive-site.xml.

<property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
</property>

回答2:

Sometimes the hive jar in spark could be different than hive version installed, to handle this scenario, You can Pass the jars and version in conf parameter to spark job submit E.g. --conf spark.sql.hive.metastore.version=2.3.0 --conf spark.sql.hive.metastore.jars=/home/apache-hive-2.3.6-bin/lib/*

回答3:

I got the similar issue due to Spark & Hive Version mis-match.Spark 2.0 points to 1.2.0 Hive version and the default Hive i was using was 0.14.0. So by passing version while starting the pyspark will resolve the issue.

pyspark --master yarn --num-executors 1 --executor-memory 512M --conf spark.sql.hive.metastore.version=0.14.0 --conf spark.sql.hive.metastore.jars=/usr/local/hive/apache-hive-0.14.0-bin/*