How does Apache Spark handles system failure when

Question

Preconditions

Let's assume Apache Spark is deployed on a hadoop cluster using YARN. Furthermore a spark execution is running. How does spark handle the situations listed below?

Cases & Questions

1. One node of the hadoop clusters fails due to a disc error. However replication is high enough and no data was lost.
   • What will happen to tasks that where running at that node?
2. One node of the hadoop clusters fails due to a disc error. Replication was not high enough and data was lost. Simply spark couldn't find a file anymore which was pre-configured as resource for the work flow.
   • How will it handle this situation?
3. During execution the primary namenode fails over.
   • Did spark automatically use the fail over namenode?
   • What happens when the secondary namenode fails as well?
4. For some reasons during a work flow the cluster is totally shut down.
   • Will spark restart with the cluster automatically?
   • Will it resume to the last "save" point during the work flow?

I know, some questions might sound odd. Anyway, I hope you can answer some or all. Thanks in advance. :)

Answer 1

Here are the answers given by the mailing list to the questions (answers where provided by Sandy Ryza of Cloudera):

1. "Spark will rerun those tasks on a different node."
2. "After a number of failed task attempts trying to read the block, Spark would pass up whatever error HDFS is returning and fail the job."
3. "Spark accesses HDFS through the normal HDFS client APIs. Under an HA configuration, these will automatically fail over to the new namenode. If no namenodes are left, the Spark job will fail."
4. Restart is part of administration and "Spark has support for checkpointing to HDFS, so you would be able to go back to the last time checkpoint was called that HDFS was available."