How to generate lots of data in Spark?
<p>Currently when I want to generate data in Spark I do something like this:</p>

<pre><code>//generates an array list of integers 0...999
final List&lt;Integer&gt; range = range(1000);
JavaRDD&lt;Data&gt; rdd = sc
   .parallelize(range)
   .mapPartitionsWithIndex(generateData(), false);
</code></pre>

<p>For large enough Range I run out of memory (500 million for example).</p>

<p>How can get around this problem?</p>