Spark - Random Number Generation
<p>I have written a method that must consider a random number to simulate a Bernoulli distribution. I am using <code>random.nextDouble</code> to generate a number between 0 and 1 then making my decision based on that value given my probability parameter.</p>

<p>My problem is that Spark is generating the same random numbers within each iteration of my for loop mapping function. I am using the <code>DataFrame</code> API. My code follows this format:</p>

<pre><code>val myClass = new MyClass()
val M = 3
val myAppSeed = 91234
val rand = new scala.util.Random(myAppSeed)

for (m &lt;- 1 to M) {
  val newDF = sqlContext.createDataFrame(myDF
    .map{row =&gt; RowFactory
      .create(row.getString(0),
        myClass.myMethod(row.getString(2), rand.nextDouble())
    }, myDF.schema)
}
</code></pre>

<p>Here is the class:</p>

<pre><code>class myClass extends Serializable {
  val q = qProb

  def myMethod(s: String, rand: Double) = {
    if (rand &lt;= q) // do something
    else // do something else
  }
}
</code></pre>

<p>I need a new random number every time <code>myMethod</code> is called. I also tried generating the number inside my method with <code>java.util.Random</code> (<code>scala.util.Random</code> v10 does not extend <code>Serializable</code>) like below, but I'm still getting the same numbers within each for loop </p>

<pre><code>val r = new java.util.Random(s.hashCode.toLong)
val rand = r.nextDouble()
</code></pre>

<p>I've done some research, and it seems this has do to with Sparks deterministic nature.</p>