Logs for hive query executed via. beeline

Question

i am running below hive coomand from beeline . Can someone please tell where can I see Map reudce logs for this ?

0: jdbc:hive2://<servername>:10003/> select a.offr_id offerID , a.offr_nm  offerNm  , b.disp_strt_ts  dispStartDt , b.disp_end_ts dispEndDt , vld_strt_ts validStartDt, vld_end_ts validEndDt from gcor_offr a, gcor_offr_dur b where a.offr_id = b.offr_id and b.disp_end_ts > '2016-09-13 00:00:00';

Answer 1

Hive uses log4j for logging. These logs are not emitted to the standard output by default but are instead captured to a log file specified by Hive's log4j properties file. By default, Hive will use hive-log4j.default in the conf/ directory of the Hive installation which writes out logs to /tmp/<userid>/hive.log and uses the WARN level.

It is often desirable to emit the logs to the standard output and/or change the logging level for debugging purposes. These can be done from the command line as follows:

$HIVE_HOME/bin/hive --hiveconf hive.root.logger=INFO,console

Answer 2

When using beeline, MapReduce logs are part of HiveServer2 log4j logs.

If your Hive install was configured by Cloudera Manager (CM), then it will typically be in /var/log/hive/hadoop-cmf-HIVE-1-HIVESERVER2-*.out on the node where HiveServer2 is running (may or may not be the same as where you are running beeline from)

Few other scenarios:

• Your Hive install was not configured by CM ? You will need to manually create log4j config file:

  • Create hive-log4j.properties config file in directory specified by HIVE_CONF_DIR environment variable. (This makes it accessible to HiveServer2 JVM classpath)
    • In this file, log location is specified by log.dir and log.file. See conf/hive-log4j.properties.template in your distribution for an example template for this file.

• You run beeline in "embedded HS2 mode" (i.e. beeline -u jdbc:hive2:// user password) ?:

  • You will customize beeline log4j (as opposed to HiveServer2 log4j).
  • Beeline log4j properties file is strictly called beeline-log4j2.properties (in versions prior to Hive 2.0, it is called beeline-log4j.properties). Needs to be created and made accessible to beeline JVM classpath via HIVE_CONF_DIR. See HIVE-10502 and HIVE-12020 for further discussion on this.

• You want to customize what HiveServer2 logs get printed on beeline stdout ?

  • This can be configured at HiveServer2 level using hive.server2.logging.operation.enabled and hive.server2.logging.operation configs.