scala – Spark在Yarn集群exitCode = 13上运行:
发布时间:2020-12-16 19:12:26 所属栏目:安全 来源:网络整理
导读:我是一个火花/纱线新手,当我在纱线集群上提交火花作业时,遇到exitCode = 13.当火花作业在本地模式下运行时,一切都很好. 我使用的命令是: /usr/hdp/current/spark-client/bin/spark-submit --class com.test.sparkTest --master yarn --deploy-mode cluster
我是一个火花/纱线新手,当我在纱线集群上提交火花作业时,遇到exitCode = 13.当火花作业在本地模式下运行时,一切都很好.
我使用的命令是: /usr/hdp/current/spark-client/bin/spark-submit --class com.test.sparkTest --master yarn --deploy-mode cluster --num-executors 40 --executor-cores 4 --driver-memory 17g --executor-memory 22g --files /usr/hdp/current/spark-client/conf/hive-site.xml /home/user/sparkTest.jar* 火花错误日志: 16/04/12 17:59:30 INFO Client: client token: N/A diagnostics: Application application_1459460037715_23007 failed 2 times due to AM Container for appattempt_1459460037715_23007_000002 exited with exitCode: 13 For more detailed output,check application tracking page:http://b-r06f2-prod.phx2.cpe.net:8088/cluster/app/application_1459460037715_23007Then,click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e40_1459460037715_23007_02_000001 Exit code: 13 Stack trace: ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) **Yarn logs** 16/04/12 23:55:35 INFO mapreduce.TableInputFormatBase: Input split length: 977 M bytes. 16/04/12 23:55:41 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 16/04/12 23:55:51 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 16/04/12 23:56:01 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 16/04/12 23:56:11 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 16/04/12 23:56:11 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x152f0b4fc0e7488 16/04/12 23:56:11 INFO zookeeper.ZooKeeper: Session: 0x152f0b4fc0e7488 closed 16/04/12 23:56:11 INFO zookeeper.ClientCnxn: EventThread shut down 16/04/12 23:56:11 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 2). 2003 bytes result sent to driver 16/04/12 23:56:11 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 82134 ms on localhost (2/3) 16/04/12 23:56:17 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x4508c270df0980316/04/12 23:56:17 INFO zookeeper.ZooKeeper: Session: 0x4508c270df09803 closed * ... 16/04/12 23:56:21 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application. 16/04/12 23:56:21 INFO yarn.ApplicationMaster: Final app status: FAILED,exitCode: 13,(reason: Timed out waiting for SparkContext.) 16/04/12 23:56:21 INFO spark.SparkContext: Invoking stop() from shutdown hook * 解决方法
您似乎已将代码中的主服务器设置为本地服务器
SparkConf.setMaster( “本地[*]”) 您必须让代码中的master取消设置,并在发出spark-submit时稍后设置它 spark-submit –master yarn-client … (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |