Tez job fails when submitting by different user

问题:

Configured Hadoop-2.6.0 HA cluster with Kerberos security. When submitting example job using tez-example-0.6.0.jar in yarn-tez framework from different user, getting the below exception

Exception

java.io.IOException: The ownership on the staging directory hdfs://clustername/tmp/staging is not as expected. It is owned by Kumar. The directory must be owned by the submitter TestUser or by TestUser

The directory has full permission but still getting the above exception. But when submitting a job using mapreduce-examples-2.6.0.jar in yarn-tez framework then the job completed successfully.

Command

yarn jar C:\Tez\tez-examples-0.6.0.jar orderedwordcount -Dmapreduce.framework.name=yarn-tez  /user/Kumar/names /user/Kumar/names1

Help would be greatly appreciated.

回答1:

Set the following in your tez-site.xml:

<property>
   <name>tez.staging-dir</name>
   <value>/tmp/${user.name}/tez/staging/</value>
</property>

The current staging directory seems to be configured to use /clustername/tmp/staging/ which ends up using the same path for multiple users causing permission conflicts.

If you use /clustername/tmp as the prefix, please be sure to pre-create the tmp dir and have 777 permissions on it.