Oozie Hive action hangs and heart beats forever

Question

I am attempting to run a Hive action through an Oozie workflow that I've created in Hue, but the action "heart beat"s forever and does not execute the Hive SQL.

I've read other posts about heart beating forever, but this one seems to be occurring at a different point, after the SQL statement has been parsed. I've checked memory on each node in the cluster, and I've verified that the task count parameters are reasonable.

Here is the hive-config.xml file:

<configuration>

<property>   
  <name>javax.jdo.option.ConnectionURL</name>   
  <value>jdbc:hive://10.1.10.250:10000/testdb</value>   
  <description>JDBC connect string</description> 
</property>

<property>   
  <name>javax.jdo.option.ConnectionDriverName</name>   
  <value>org.apache.hadoop.hive.jdbc.HiveDriver</value>   
  <description>JDBC driver</description> 
</property>

</configuration>

I know that the Hive connection is working, because the action fails if provided with either a bad SQL statement, a bad URL, or a bad driver name.

Here is the action stdout log:

  [...truncated]

  =================================================================

  >>> Invoking Hive command line now >>>

  4283 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
  4339 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
  4354 [main] INFO  hive.ql.parse.ParseDriver  - Parsing command: create table testdb.temp99 (col1 int)
  4665 [main] INFO  hive.ql.parse.ParseDriver  - Parse Completed
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=parse start=1418968298270 end=1418968298598 duration=328 from=org.apache.hadoop.hive.ql.Driver>
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
  4733 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Starting Semantic Analysis
  4735 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Creating table testdb.temp99 position=13
  4760 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Semantic Analysis Completed
  4775 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=semanticAnalyze start=1418968298598 end=1418968298706 duration=108 from=org.apache.hadoop.hive.ql.Driver>
  4784 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
  4784 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=compile start=1418968298215 end=1418968298715 duration=500 from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Concurrency mode is disabled, not creating a lock manager
  4785 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Starting command: create table testdb.temp99 (col1 int)
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=TimeToSubmit start=1418968298215 end=1418968298723 duration=508 from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
  4815 [main] INFO  hive.ql.exec.DDLTask  - Default to LazySimpleSerDe for table testdb.temp99
  4935 [main] INFO  org.apache.hadoop.hive.metastore.HiveMetaStore  - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  4959 [main] INFO  org.apache.hadoop.hive.metastore.ObjectStore  - ObjectStore, initialize called
  5261 [main] INFO  DataNucleus.Persistence  - Property datanucleus.cache.level2 unknown - will be ignored
  Heart beat
  Heart beat
  [...forever...]

Why does the workflow heart beat at this point in the log rather than continuing?

ADDENDUM:

The Oozie workflow associated with this Hive action is:

<workflow-app name="Hive-copy" xmlns="uri:oozie:workflow:0.4">
    <start to="Hive"/>
    <action name="Hive">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <job-xml>/user/test/hive-config.xml</job-xml>
            <script>/user/test/test.sql</script>
            <file>hive-config.xml#hive-config.xml</file>
        </hive>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

Answer 1

This issue is related to nodeManager. Try to set the following property in the mapred-site.xml file fixes the issue:

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>

It was similar to this issue : https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/v0BHtQ0hlBg