Keras model pyspark error

Question

I have a keras model that has been pickled as described in the following blog.

http://zachmoshe.com/2017/04/03/pickling-keras-models.html

What's strange is that, when I ran the model on an html file when read from python as open(filename), it worked as expected. But when running it on a file when read from pyspark, I am getting the following error:

AttributeError("'Model' object has no attribute '_feed_input_names'",)

Answer 1

You have to run make_keras_picklable() on each worker as well. Otherwise, the __setstate__ method of Model object on worker node is not updated and thus will not be deserialized as expected.