Spark: How to write org.apache.spark.rdd.RDD[java.io.ByteArrayOutputStream]
<p>I have an RDD that has the signature</p>

<pre><code>org.apache.spark.rdd.RDD[java.io.ByteArrayOutputStream]
</code></pre>

<p>In this RDD, each row has its own partition. </p>

<p>This ByteArrayOutputStream is zip output. I am applying some processing on the data in each partition and i want to export the processed data from each partition as a single zip file. What is the best way to export each Row in the final RDD as one file per row on hdfs?</p>

<p>If you are interested in knowing how I ended up with such an Rdd.</p>

<pre><code>val npyData = transformedTopData.select("tokenIDF", "topLevelId").rdd.repartition(2).mapPartitions(x =&gt; {
      val vectors = for {
        row &lt;- x
      } yield {
        row.getAs[Vector](0)
      }
      Seq(ml2npyCSR(vectors.toSeq).zipOut)
    }.iterator)
</code></pre>

<p>EDIT: Count works perfectly fine</p>

<pre><code>scala&gt; npyData.count()
res9: Long = 2 
</code></pre>