What to do with a Spark Job that hangs on the last task?
<p>I am using <a href="https://github.com/alitouka/spark_dbscan" rel="nofollow">https://github.com/alitouka/spark_dbscan</a>, and to determine parameters, I am using the utility class they supply, org.alitouka.spark.dbscan.exploratoryAnalysis.DistanceToNearestNeighborDriver.</p>

<p>I am on a 10 node cluster with one machine with 8 cores and 32G of memory and nine machines with 6 cores and 16G of memory.</p>

<p>I have 442M of data, which seems like it would be a joke, but the job stalls at the last stage.</p>

<p>It was stuck in Scheduler Delay for 10 hours overnight, and I have tried a number of things for the last couple days, but nothing seems to be helping.</p>

<p>I have tried:</p>

<ul>
<li>Increasing heap sizes and numbers of cores</li>
<li>More/less executors with different amounts of resources.</li>
<li>Kyro Serialization</li>
<li>FAIR Scheduling</li>
</ul>

<p>The Spark Version is 1.4.1</p>

<p>The logs are full of standard fair, nothing like an exception or even interesting [INFO] lines.</p>

<p>Here is the script I am using: <a href="https://gist.github.com/isaacsanders/660f480810fbc07d4df2" rel="nofollow">https://gist.github.com/isaacsanders/660f480810fbc07d4df2</a></p>

<p>Hadoop is: HDP 2.3.2.0-2950</p>

<p>Here is a gist (pastebin) of my versions en masse and a stacktrace: <a href="https://gist.github.com/isaacsanders/2e59131758469097651b" rel="nofollow">https://gist.github.com/isaacsanders/2e59131758469097651b</a></p>

<p><a href="https://github.com/alitouka/spark_dbscan/blob/master/src/src/main/scala/org/alitouka/spark/dbscan/exploratoryAnalysis/DistanceToNearestNeighborDriver.scala" rel="nofollow">https://github.com/alitouka/spark_dbscan/blob/master/src/src/main/scala/org/alitouka/spark/dbscan/exploratoryAnalysis/DistanceToNearestNeighborDriver.scala</a></p>