Why only one core is taking all the load , how to

2019-02-20 16:51发布

I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.

Below it the my spark cluster configuration

Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:

$SPARK_HOME/bin/spark-submit 
--master yarn 
--deploy-mode cluster 
--name app 
--class Driver 
--executor-cores 3 
--executor-memory 8g 
--num-executors 10 
--driver-cores 2 
--driver-memory 10g 
--conf spark.shuffle.service.enabled=true 
--conf spark.dynamicAllocation.enabled=false 
--conf spark.task.maxFailures=8  
--conf spark.ui.killEnabled=false 
 oracle2c.jar 

I have made the Spark data frame partitions 30 as below

+--------------------+-------+
|SPARK_PARTITION_ID()|  count|
+--------------------+-------+
|                  22|6687176|
|                  29|6687175|
|                   8|6687176|
|                  21|6687176|
|                  23|6687176|
|                   5|6687176|
|                   9|6687176|
|                  17|6687176|
|                  26|6687175|
|                  27|6687175|
|                   4|6687176|
|                  10|6687176|
|                  18|6687176|
|                  25|6687175|
|                   1|6687176|
|                  20|6687176|
|                  12|6687176|
|                  28|6687175|
|                  24|6687176|
|                  15|6687176|
|                  14|6687176|
|                   6|6687176|
|                  19|6687176|
|                   0|6687175|
|                   3|6687176|
|                  11|6687176|
|                   2|6687176|
|                   7|6687176|
|                  13|6687176|
|                  16|6687176|
+--------------------+-------+

But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?

enter image description here

What more configuration parameters I need to add to make all the cores take the load ?

0条回答
登录 后发表回答