How to use mapPartitions in Spark Scala?
<p>I have DocsRDD : RDD[String, String]</p>

<pre><code>val DocsRDD = sc.wholeTextFiles("myDirectory/*" , 2)
</code></pre>

<p>DocsRDD:</p>

<pre><code>Doc1.txt , bla bla bla .....\n bla bla bla \n bla ... bla
Doc2.txt , bla bla bla .....bla \n bla bla \n bla ... bla
Doc3.txt , bla bla bla .....\n bla bla bla \n bla ... bla
Doc4.txt , bla bla \n  .....\n bla bla bla bla \n ... bla
</code></pre>

<p>Is there an efficient, elegant way to extract n-grams from these with mapPartitions?
So far i have tried everything, i have read everything i could find at least 5 times over and over about mapPartitions but i still cannot understand how to use it! It seems waaay too difficult to manipulate.
In short i want :</p>

<pre><code>val NGramsRDD = DocsRDD.map(x =&gt; (x._1 , x._2.sliding(n) ) )
</code></pre>

<p>but efficiently with mapPartitions.
My basic misunderstanding of mapPartitions is :</p>

<p>OneDocRDD : RDD[String]</p>

<pre><code> val OneDocRDD = sc.textFile("myDoc1.txt" , 2)
                   .mapPartitions(s1 : Iterator[String] =&gt; s2 : Iterator[String])
</code></pre>

<p>I Cannot understand this! From when s1 was Iterator[String]? s1 is String after sc.textfile.</p>

<p>Alright my second question is : Will mapPartitions improve my overcome against map in this situation?</p>

<p>Last but not Least important: 
can f() be :</p>

<pre><code>     f(Iterator[String]) : Iterator[Something else?]
</code></pre>