How to generate random vector in Spark
<p>I want to generate random vectors with norm 1 in Spark.</p>

<p>Since the vector could be very large, I want it to be distributed, And since data in RDD has no order, I want to store the vector in the form of RDD[(Int, Double)], because I also need to use this vector to do some matrix-vector multiplication.</p>

<p>So how could I generate this kind of vector?</p>

<p>Here is my plan for now:</p>

<pre><code>val v = normalRDD(sc, n, NUM_NODE)
val mod = GetMod(v)       // Get the modularity of v
val res = v.map(x =&gt; x / mod)
val arr:Array[Double] = res.toArray()

var tuples = new List[(Int, Double)]()
for (i &lt;- 0 to (arr.length - 1)) {
  tuples = (i, arr(i)) :: tuples
}
// Get the entries and length of the vector.
entries = sc.parallelize(tuples)
length = arr.length
</code></pre>

<p>I think it not elegant enough because it goes through a "distributed -> single node -> distributed" process.</p>

<p>Is there any way better? Thanks:D</p>