External hive metastore for EMR

问题:

I am creating a EMR cluster with default hive meta store , after which i am overriding the hive-site.xml with some property which are pointing the aws rds instance as hive metastore , everything is fine , but after restarting the hive server , i am not able to use RDS as hive metastore. It is still usin the default hive metastore created by EMR.

回答1:

You can override the default configurations for applications by supplying a configuration object for applications when you create a cluster. The configuration object is referenced as a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties are the settings you want to change in that file. You can specify multiple classifications for multiple applications in a single JSON object.

For overriding hive-site.xml with your external mysql metastore information, create a configuration file called hiveConfiguration.json containing edits to hive-site.xml:

[
    {
      "Classification": "hive-site",
      "Properties": {
        "javax.jdo.option.ConnectionURL": "jdbc:mysql:\/\/hostname:3306\/hive?createDatabaseIfNotExist=true",
        "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",
        "javax.jdo.option.ConnectionUserName": "username",
        "javax.jdo.option.ConnectionPassword": "password"
      }
    }
]

Use hiveConfiguration.json with the following AWS CLI command to create the cluster:

aws emr create-cluster --release-label emr-5.11.0 --instance-type m3.xlarge --instance-count 2 \
--applications Name=Hive --configurations ./hiveConfiguration.json --use-default-roles

Reference :

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-external.html

回答2:

Why do you restart Hive-server2 ? If you are changing properties like hive.metastore.* , then you will need to restart Hive Metastore daemon instead, which on EMR is part of Hcatalog. Actually, you might not need to even restart anything for just updating the metastore DB. You can just run offline schematool -initSchema to point to the new DB. see https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool