Spark "Executor heartbeat timed out"
<p>I have a simple reproducible Spark error. (Spark 2.0 + Amazon EMR 5.0 FYI)</p>

<pre><code>def row_parse_function():
    # Custom row parsing function. Details omitted.
    return pyspark.sql.types.Row(...)


if __name__ == "__main__"
    spark_context = build_spark_context("max value bug isolation")
    spark_sql_context = SQLContext(spark_context)

    full_source_path = "s3a://my-bucket/ten_gb_data_file.txt.gz"

    # Tried changing partition parameter to no effect.
    raw_rdd = spark_context.textFile(full_source_path, 5000)
    row_rdd = raw_rdd.map(row_parse_function).filter(bool)
    data_frame = spark_sql_context.createDataFrame(row_rdd, AttribPixelMergedStructType)
    # Tried removing and chaning this repartition call to no effect.
    data_frame.repartition(5000)
    # Removing this cache call makes this small sample work.
    data_frame.cache()
    data_frame_count = data_frame.count()
</code></pre>

<p>This fails with:</p>

<pre><code>ExecutorLostFailure (executor 5 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 169068 ms
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
</code></pre>

<p>I know the heartbeat timed out error usually means the worker died, typically due to lack of memory. How do I resolve this?</p>