Hive error on all queries: Couldn't find log a

Question

Getting an error on all hive queries running on Hadoop.

Simply running a select statement from a table that exists and getting the following results:

Fetching results ran into the following error(s):

Couldn't find log associated with operation handle: 
OperationHandle[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]

And the accompaning logs for the query are:

Bad status for request TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='%\x1d\x0f\x11\xce\xf4B\xd6\xb8\xb5\x8ddv\xceKV', guid="\xbc5\xd9|\xfa\x13J'\x86\x8e4\xb3\x87\xa0\xf4\x05")), orientation=4, maxRows=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=bc35d97c-fa13-4a27-868e-34b387a0f405]", sqlState=None, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=bc35d97c-fa13-4a27-868e-34b387a0f405]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None) Bad status for request TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='c\xdf\xfe\xb7\x8a\x10D\xea\xbaD\x93}B\xb5\xc3\xe7', guid='&=\xf9\xc3\xc0vEO\xa0%#\xf5\xb9\x19\xe9W')), orientation=0, maxRows=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]", sqlState=None, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None) Bad status for request TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='c\xdf\xfe\xb7\x8a\x10D\xea\xbaD\x93}B\xb5\xc3\xe7', guid='&=\xf9\xc3\xc0vEO\xa0%#\xf5\xb9\x19\xe9W')), orientation=0, maxRows=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]", sqlState=None, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None)

Answer 1

If you're running into this issue due to a shell script or java program, the first thing you need to check is the connection string, I have faced similar issue few months back, when in my script I was running the queries in such manner:

beeline -u  jdbc:hive2:// -e "show databases;"

This was due to missing of proper Hive Connection string, once I corrected the connection string and ran the query, it ran without any issues:

beeline -u  jdbc:hive2://server.enterprise.org:10000/default -e "show databases;"

This answer is only specific to the developers who are using a script or code, for Hue/Cloudera distro, kindly refer to the above answers.

Cheers!

Answer 2

If you are seeing this in a Cloudera distro when you use Hue, then add the following to the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml in Cloudera Manager (or straight to the hive-site.xml for other distros):

<property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
</property>

Answer 3

If you are using AWS EMR, then add the following to the hive-site.xml file at /etc/hive/conf.dist/hive-site.xml:

<property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
</property>

Then log into the master node and restart hue and hive server

sudo stop hive-server2
sudo stop hue
sudo start hue
sudo start hive-server2

This fixed it for me (tested in Release label emr-4.6.0).

Answer 4

This error can happen with the i.e(“hive.server2.logging.operation.log.location”) is pointing to a location that does not exist to write logs.

Look at the code from OperationManager.java

OperationLog operationLog = getOperation(opHandle).getOperationLog();
if (operationLog == null) {
  throw new HiveSQLException("Couldn't find log associated with operation handle: " + opHandle);
}

Answer 5

In Windows 8.1, I were also facing the same issue part inside hive query part -

org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=230485c4-a1be-4a17-94c8-a1ebce426319]

thus, I went to the %HIVE_HOME%/conf/hive-site.xml and changed the property name hive.server2.logging.operation.log.location earlier value ${system:user.name}/operation_logs to like below said -

  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/tmp/${user.name}/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

and also replaced all the values which contains the ${system:user.name} to /tmp/${user.name} and this brings me up by the job logging issue in hive, thanks.