Do I have to create /tmp and /user/hive/warehouse?

Question

From https://cwiki.apache.org/confluence/display/Hive/GettingStarted

Running Hive

Hive uses Hadoop, so:

• you must have Hadoop in your path OR
• export HADOOP_HOME=

In addition, you must use below HDFS commands to create /tmp and /user/hive/warehouse (aka hive.metastore.warehouse.dir) and set them chmod g+w before you can create a table in Hive.

  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

I can't run the two -mkdir commands:

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
mkdir: `/tmp': File exists

`/tmp' already exists on my local ext4 file system in my Ubuntu.

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
mkdir: `/user/hive/warehouse': No such file or directory

There is no /usr/hive/warehouse in my local ext4 file system in my Ubuntu.

So what is the purpose of running -mkdir to create the two directories?

Answer 1

mkdir: `/tmp': File exists

This folder is already present in hdfs. So you are getting 'File exists' error while running mkdir command.

You can check with list command: hadoop fs -ls /

mkdir: `/user/hive/warehouse': No such file or directory

mkdir command creates new directory however does not create parent directories. If any of the parent directories don’t exist in the path, you will get 'No such file or directory' error. In this case either 'user' or 'hive' directory does not exists.

You can check with list command: hadoop fs -ls /user

Running 'mkdir' command with '-p' option creates all missing parent directories.

hadoop fs -mkdir -p /user/hive/warehouse 

Both '/tmp' and '/user/hive/warehouse' directories are created in hdfs file system. You cannot see these folders using local file system commands.

`/tmp' already exists on my local ext4 file system in my Ubuntu.

This /tmp folder is maintained by your operating system and used to hold temporary files. Not related to hdfs.

To list folders in hdfs,

hadoop fs -ls /user
hadoop fs -ls /
hadoop fs -ls /user/hive/