How can I parallelize multiple Datasets in Spark?

Question

I have a Spark 2.1 job where I maintain multiple Dataset objects/RDD's that represent different queries over our underlying Hive/HDFS datastore. I've noticed that if I simply iterate over the List of Datasets, they execute one at a time. Each individual query operates in parallel, but I feel that we are not maximizing our resources by not running the different datasets in parallel as well.

There doesn't seem to be a lot out there regarding doing this, as most questions appear to be around parallelizing a single RDD or Dataset, not parallelizing multiple within the same job.

Is this inadvisable for some reason? Can I just use a executor service, thread pool, or futures to do this?

Thanks!

Answer 1

Yes you can use multithreading in the driver code, but normally this does not increase performance, unless your queries operate on very skewed data and/or cannot be parallelized well enough to fully utilize the resources.

You can do something like that:

val datasets : Seq[Dataset[_]] = ???

datasets
  .par // transform to parallel Seq
  .foreach(ds => ds.write.saveAsTable(...)