Can't connect to CFS node

Question

I removed (or decommisioned, can't remember) a DSE analytics node (with IP 10.14.5.50) a couple of months ago. When I now try to execute a dse shark (CREATE TABLE ccc AS SELECT ...) query I now receiving:

15/01/22 13:23:17 ERROR parse.SharkSemanticAnalyzer: org.apache.hadoop.hive.ql.parse.SemanticException: 0:0 Error creating temporary folder on: cfs://10.14.5.50/user/hive/warehouse/mykeyspace.db. Error encountered near token 'TOK_TMP_FILE'
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1256)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1053)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8342)
    at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:105)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
    at shark.SharkDriver.compile(SharkDriver.scala:215)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
    at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:347)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
    at shark.SharkCliDriver$.main(SharkCliDriver.scala:240)
    at shark.SharkCliDriver.main(SharkCliDriver.scala)
Caused by: java.lang.RuntimeException: java.io.IOException: Error connecting to node 10.14.5.50:9160 with strategy STICKY.
    at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:216)
    at org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:270)
    at org.apache.hadoop.hive.ql.Context.getExternalTmpFileURI(Context.java:363)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1253)
    ... 12 more

I guess the above error is due to my keyspace referring to the old node:

shark> DESCRIBE DATABASE mykeyspace;
OK
mykeyspace      cfs://10.14.5.50/user/hive/warehouse/mykeyspace.db
Time taken: 0.997 seconds

Is there any way for me to fix this incorrect database path?

Tried (but failed) workaround to recreate the database: In cqlsh I created a keyspace thekeyspace and added a table thetable. I the opened up dse hive (and noticed that DESCRIBE DATABASE thekeyspace is giving me a correct cfs path). However, I am unable to drop the the database using DROP DATABASE thekeyspace.

Additional information:

• I have no external tables in my keyspace.
• Making the SELECT against the tables works.
• Setting -hiveconf cassandra.host=WORKING_NODE_IP does not help.
• The following commands return proper IP:s (ie. not X.X.X.50):
  • dsetool listjt
  • dsetool jobtracker
  • dsetool sparkmaster
• I am getting the same error when I execute the query using dse hive.
• No Shark variable is referring to X.X.X.50 when I execute set; in its REPL.
• I am running DSE 4.5.

Answer 1

Stumbled across this page that says you need to TRUNCATE "HiveMetaStore"."MetaStore" (in cqlsh) after removing Hive nodes. That did the trick.