How to make a special RDD in spark application
<p>i Have a RDD(K, Iterable[V])</p>

<p>how can convert it to RDD(K, RDD[V]) without collect in driver</p>

<p>-------------------edited again----------------------------------------------</p>

<pre><code>val dataList = DataLoader.loadTrainTestData(hiveContext.sql(sampleDataHql)).collect().map(ds =&gt; (ds._1, sc.parallelize(ds._2.toSeq)))

//Train Model and Test
val resultData = sc.parallelize(dataList).map{ ds =&gt;
  val remark = ds._1
  val data = ds._2.randomSplit(Array(0.6, 0.4), seed = 11L)

  val model = new LogisticRegressionWithLBFGS().setNumClasses(2).setIntercept(true).run(data(0))

  val trainAUC = ModelTester.getAUC(data(1), model)
  val modelWeight = model.weights.toArray.map(_.toString).reduce(_ + "_" + _)

  modelWeight + "|" + model.intercept.toFloat + "|" + model.numClasses+ "|" + model.numFeatures + "|" + trainAUC.toFloat + "|" + remark
}
</code></pre>

<p>i haved created a nested RDD, but it will collect and re-parallelize in driver, take up too much memory.</p>

<p>-------------------edited again----------------------------------------------</p>

<p>or how can i convert <strong>RDD(K, Iterable[V])</strong> to <strong>Array(K, RDD[V])</strong> without collect on driver</p>

<p>or what is the better way to run <strong>multi-dataset</strong></p>

<p>-------------------edited again----------------------------------------------</p>

<p>nested RDD is not allowed indeed! thanks yours!</p>

<p>and if  there is a way to create <strong>Array(K, RDD[V])</strong> without collect in driver?</p>