reuse SparkHadoopWriter in rdd.saveAsHadoopFile
<p>The Spark rdd.saveAsHadoopFile is very wasteful in that it generates a new SparkHadoopWriter on every write. We have a use case where the Spark job is backed up simply because gc on these Writers can not keep up with incoming stream.</p>

<p>Has any one done a reimplementation of this method where the SparkHadoopWriter is reused based on let's say which bucket the write target is. If I go in that direction, is there any architectural reason why the effort would fail?</p>