Can beeline and spark-shell show different databas

Question

I have installed hadoop 2.8.1 on ubuntu and then installed spark-2.2.0-bin-hadoop2.7 on it. Firstly when I created database through spark-shell and tryied to access it through java JDBC program I was seing no table exists. Then I used beeline and observed that the databases do not exists here too. I created databases through beeline. Why does spark-shell and beeline show different DBs?

They probably should show the same. I tried the plain JDBC program which connects hive2 and fetched the tables and observed that sometimes it show tables I creatd through spark-shell and sometimes shows that of created through beeline...Please help. The same is happening sometimes with beeline too..

Answer 1

This is probably because your Spark installation is not configured properly to access your Hive warehouse.

In such case, Spark-SQL is designed to setup its own local warehouse and revert to standalone mode. This is intended to ease adoption for non-Hive users.

To troubleshoot this, you should:

1. Refer to the official documentation.
2. read the logs and look for anything related to 'hive' or 'metastore' to understand what happens

3. make sure that Spark has access to the hive-site.xml configuration file. You can for instance set up a symlink (be sure to check the paths first)

   ln -s /etc/hive/conf/hive-site.xml    /etc/spark/conf/hive-site.xml
   

4. make sure that your Spark installation has access to the Hive jars (check $SPARK_HOME/lib)

5. make sure you have enabled Hive support with something like this:

   SparkSession.builder.enableHiveSupport().getOrCreate()
   

Hope this helps.