Spark Stand Alone - Last Stage saveAsTextFile takes many hours using very little resources to write CSV part files
<p>We run Spark in Standalone mode with 3 nodes on a 240GB "large" EC2 box to merge three CSV files read into DataFrames to JavaRDDs into output CSV part files on S3 using s3a.</p>

<p>We can see from the Spark UI, the first stages reading and merging to produce the final JavaRDD run at 100% CPU as expected, but the final stage writing out as CSV files using <code>saveAsTextFile at package.scala:179</code> gets "stuck" for many hours on 2 of the 3 nodes with 2 of the 32 tasks taking hours (box is at 6% CPU, memory 86%, Network IO 15kb/s, Disk IO 0 for the entire period).</p>

<p>We are reading and writing uncompressed CSV (we found uncompressed was much faster than gzipped CSV) with re partition 16 on each of the three input DataFrames and not coleaseing the write.</p>

<p>Would appreciate any hints what we can investigate as to why the final stage takes so many hours doing very little on 2 of the 3 nodes in our standalone local cluster.</p>

<p>Many thanks</p>

<p>--- UPDATE ---</p>

<p>I tried writing to local disk rather than s3a and the symptoms are the same - 2 of the 32 tasks in the final stage <code>saveAsTextFile</code> get "stuck" for hours:</p>

<p><a href="https://i.stack.imgur.com/hqxHO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hqxHO.png" alt="enter image description here"></a></p>