scala – 使用带有Spark on Yarn的typesafe配置

读取配置的代码如下所示：

ConfigFactory.load().getConfig("com.mycompany")

现在我没有将application.conf作为我的超级jar的一部分进行组装,因为我想将该文件作为外部文件传递

我想使用的外部application.conf的内容如下所示：

com.mycompany {
  //configurations my program needs
}

此application.conf文件存在于我的本地计算机文件系统上(而不是HDFS上)

我正在使用Spark 1.6.1和Yarn

这是我的spark-submit命令的样子：

LOG4J_FULL_PATH=/log4j-path
ROOT_DIR=/application.conf-path

    /opt/deploy/spark/bin/spark-submit 
    --class com.mycompany.Main 
    --master yarn 
    --deploy-mode cluster 
    --files $ROOT_DIR/application.conf 
    --files $LOG4J_FULL_PATH/log4j.xml 
    --conf spark.executor.extraClassPath="-Dconfig.file=file:application.conf" 
    --driver-class-path $ROOT_DIR/application.conf 
    --verbose 
    /opt/deploy/lal-ml.jar

我收到的例外是：

2016-11-09 12:32:14 ERROR ApplicationMaster:95 - User class threw exception: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'com'
com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'com'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
    at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:218)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:224)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:33)
    at com.mycompany.Main$.main(Main.scala:36)
    at com.mycompany.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)

所以我的问题是：有没有人知道如何加载一个外部类型安全的application.conf文件,该文件位于我的本地机器上,带有spark-submit和yarn？

我在How to add a typesafe config file which is located on HDFS to spark-submit (cluster-mode)?和Typesafe Config in Spark以及How to pass -D parameter or environment variable to Spark job?尝试了一些解决方案,但没有任何效果

我会很感激解决这个问题的任何方向

提前致谢