I use spark 1.6.1.
My spark application reads more than 10000 parquet files stored in s3.
val df = sqlContext.read.option("mergeSchema", "true").parquet(myPaths: _*)
myPaths
is an Array[String]
that contains the paths of the 10000 parquet files. Each path is like this s3n://bucketname/blahblah.parquet
Spark warns message like below.
WARN TaskSetManager: Stage 4 contains a task of very large size (108KB). The maximum recommended task size is 100KB.
Spark has managed to run and finish the job anyway but I guess this can slow down spark processing job.
Does anybody has a good suggestion about this problem?
The issue is that your dataset is not evenly distributed across partitions and hence some partitions have more data than others (and so some tasks compute larger results).
By default Spark SQL assumes 200 partitions using
spark.sql.shuffle.partitions
property (see Other Configuration Options):A solution is to
coalesce
orrepartition
your Dataset after you've read parquet files (and before executing an action).Use
explain
or web UI to review execution plans.The warning gives you a hint to optimize your query so the more effective result fetch is used (see TaskSetManager).
With the warning TaskScheduler (that runs on the driver) will fetch the result values using the less effective approach
IndirectTaskResult
(as you can see in the code).