spark on yarn, Container exited with a non-zero exit code 143
<p>I am using HDP 2.5, running <em>spark-submit</em> as yarn cluster mode.</p>

<p>I have tried to generate data using dataframe cross join.
i.e </p>

<pre><code>val generatedData = df1.join(df2).join(df3).join(df4)
generatedData.saveAsTable(...)....
</code></pre>

<p><strong>df1</strong> storage level is MEMORY_AND_DISK</p>

<p><strong>df2,df3,df4</strong> storage level is MEMORY_ONLY</p>

<p>df1 has much more records i.e 5 million while df2 to df4 has at most 100 records.
doing so my explain plain would result with better performance using <em>BroadcastNestedLoopJoin</em> explain plan.</p>

<p>for some reason it always fail. I don't know how can I debug it and where the memory explode. </p>

<p>Error log output:</p>

<pre><code>16/12/06 19:44:08 WARN YarnAllocator: Container marked as failed: container_e33_1480922439133_0845_02_000002 on host: hdp4. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/12/06 19:44:08 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_e33_1480922439133_0845_02_000002 on host: hdp4. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/12/06 19:44:08 ERROR YarnClusterScheduler: Lost executor 1 on hdp4: Container marked as failed: container_e33_1480922439133_0845_02_000002 on host: hdp4. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/12/06 19:44:08 WARN TaskSetManager: Lost task 1.0 in stage 12.0 (TID 19, hdp4): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container marked as failed: container_e33_1480922439133_0845_02_000002 on host: hdp4. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal
</code></pre>

<p>I didn't see any WARN or ERROR logs before this error.
What is the problem? where should I look for the memory consumption? 
I cannot see anything on the <strong><em>Storage</em></strong> tab of SparkUI. 
the log was taken from <strong>yarn resource manager UI</strong> on HDP 2.5</p>

<p><strong>EDIT</strong>
looking at the container log, it seems like it's a <code>java.lang.OutOfMemoryError: GC overhead limit exceeded</code></p>

<p>I know how to increase the memory, but I don't have any memory anymore.
How can I do a cartesian / product join with 4 Dataframes without getting this error.</p>