SparkException: Python worker did not connect back

I am trying to submit a Python job to a 2 worker node Spark cluster but am persistently seeing the following problem, which eventually causes spark-submit to fail:

15/07/04 21:30:40 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID
 2, workernode0.rhom-spark.b9.internal.cloudapp.net):    
org.apache.spark.SparkException: Python worker did not connect back in time
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:135)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:64)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:102)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:278)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
    at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:305)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:278)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:56)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Accept timed out
    at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)

    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    at java.net.ServerSocket.implAccept(ServerSocket.java:530)
    at java.net.ServerSocket.accept(ServerSocket.java:498)
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:130)
    ... 15 more

I'm submitting the job with the following

spark-submit --master yarn --py-files tile.py --num-executors 1 --executor-memory 2g main.py

Any ideas?

So this happens when the python worker process fails to connect to the spark executor JVM. Spark uses sockets to communicate with the worker process. There are a large number of reasons why this could happen, and the exact specific details will likely be in the logs on the executor/worker machines.