Spark : How to use mapPartition and create/close connection per partition
<p>So, I want to do certain operations on my spark DataFrame, write them to DB and create another DataFrame at the end. It looks like this : </p>

<pre><code>import sqlContext.implicits._

val newDF = myDF.mapPartitions(
  iterator =&gt; {
    val conn = new DbConnection
    iterator.map(
       row =&gt; {
         addRowToBatch(row)
         convertRowToObject(row)
     })
    conn.writeTheBatchToDB()
    conn.close()
  })
  .toDF()
</code></pre>

<p>This gives me an error as mapPartitions expects return type of <code>Iterator[NotInferedR]</code>, but here it is <code>Unit</code>. I know this is possible with forEachPartition, but I'd like to do the mapping also. Doing it separate would be an overhead (extra spark job). What to do?</p>

<p>Thanks!</p>