I use spark 1.6.1.
My spark application reads more than 10000 parquet files stored in s3.
val df = sqlContext.read.option("mergeSchema", "true").parquet(myPaths: _*)
myPaths is an Array[String] that contains the paths of the 10000 parquet files. Each path is like this s3n://bucketname/blahblah.parquet
Spark warns message like below.
WARN TaskSetManager: Stage 4 contains a task of very large size
(108KB). The maximum recommended task size is 100KB.
Spark has managed to run and finish the job anyway but I guess this can slow down spark processing job.
Does anybody has a good suggestion about this problem?
The issue is that your dataset is not evenly distributed across partitions and hence some partitions have more data than others (and so some tasks compute larger results).
By default Spark SQL assumes 200 partitions using spark.sql.shuffle.partitions property (see Other Configuration Options):
spark.sql.shuffle.partitions (default: 200) Configures the number of partitions to use when shuffling data for joins or aggregations.
A solution is to coalesce or repartition your Dataset after you've read parquet files (and before executing an action).
Use explain or web UI to review execution plans.
The warning gives you a hint to optimize your query so the more effective result fetch is used (see TaskSetManager).
With the warning TaskScheduler (that runs on the driver) will fetch the result values using the less effective approach IndirectTaskResult (as you can see in the code).
