Submit an application property file with Spark typ

Question

Please, I need your help, I'm trying to submit an external configuration file for my spark application using typesafe config.

I'm loading the application.conf file in my application code like this:

  lazy val conf = ConfigFactory.load()

File content

  ingestion{
  process {
    value = "sas"
  }
  sas {
    origin{
      value = "/route"
    }
    destination{
      value = "/route"
    }
    extension{
      value = ".sas7bdat"
    }
    file{
      value = "mytable"
    }
    month{
      value = "201010,201011"
    }
    table{
      value = "tbl"
    }
  }
}

My spark submit is

spark2-submit --class com.antonio.Main --master yarn --deploy-mode client --driver-memory 10G --driver-cores 8 --executor-memory 13G --executor-cores 4 --num-executors 10 --verbose  --files properties.conf /home/user/ingestion-1.0-SNAPSHOT-jar-with-dependencies.jar --files application.conf

But for some reason, I'm receiving

com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'ingestion'

Everything looks configured correctly ?? Have I missed something.

thanks,

Antonio

Answer 1

Your application.conf by default must be present at the root of classpath for ConfigFactory.load() to find it. Alternatively, you can modify where to find the application.conf file through system properties. Therefore, your options are as follows.

First alternative is, add the root directory of the job to classpath:

spark2-submit ... \
    --conf spark.driver.extraClassPath=./  \
    --conf spark.executor.extraClassPath=./  \    // if you need to load config at executors
    ...

Keep the --files option as is. Note that if you run your job in the client mode, you must pass the proper path to where application.conf is located on the driver machine to the spark.driver.extraClassPath option.

Second alternative is (and I think this one is superior), you can use the config.file system property to affect where ConfigFactory.load() looks for the config file:

spark2-submit ... \
    --conf spark.driver.extraJavaOptions=-Dconfig.file=./application.conf \
    --conf spark.executor.extraJavaOptions=-Dconfig.file=./application.conf \
    ...

Remarks about loading config on executors and keeping the --files option also apply here.