Is IT possible to use DEAP ( http://deap.readthedocs.io/en/master/) with a spark cluster to map the fitness evaluation function. I would like to run a GA but the fitness function is rather long and I was planning on distributing it on a spark cluster.
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试):
问题:
回答1:
You should look at the Using Multiple Processors section in the DEAP documentation and at this example. They explain how to replace the map function in the DEAP toolbox by a map function of your choice.
To use pyspark to map the fitness evaluation function, you could do something like that:
from pyspark import SparkContext
sc = SparkContext(appName="DEAP")
def sparkMap(algorithm, population):
return sc.parallelize(population).map(algorithm)
toolbox.register("map", sparkMap)