Kinit with Spark when connecting to Hive

问题:

I am trying to connect to Hive(hadoop cluster has kerberos authentication) from Spark which is Standalone.

Can someone let me know how to do kinit in spark program i could connect to hive?

UPDATE:My Spark is on different cluster from Hadoop

回答1:

Assuming you have a spark-shell open and you don't want to exit, and then re-kinit you could do something like this:

import java.lang.ProcessBuilder
import java.io.PrintWriter

//resets your kerberos login
val p1 = Runtime.getRuntime.exec("kdestroy")
p1.waitFor
//executes kinit, 
val p = Runtime.getRuntime.exec("kinit")
val stdin = p.getOutputStream
val pw =new PrintWriter(stdin)
//val pwd = get_password() //get_password() is a function to get your password from a file, or wherever
pw.println(pwd) // you could put your password here , but plain text passwords are generally frowned upon
pw.close
p.waitFor

回答2:

you should run the kinit command before init the spark-shell or submit a spark app (spark-submit). The other option is get the kerberos ticket using a keytab and run your spark program like this

bash -c "kinit -kt /path/to/key/mykey.keytab [e-mail]; spark-shell"

regards,