Updating a PySpark dataframe column with an RDD of tuples
<p>I am working with data where user info is string. I would like to assign unique integer values to those strings. </p>

<p>I was somewhat following this stack overflow post <a href="https://stackoverflow.com/questions/27772769/how-to-use-mllib-recommendation-if-the-user-ids-are-string-instead-of-contiguous">here</a>. I am using the expression below to have an RDD of tuples:</p>

<pre><code>user = data.map(lambda x:x[0]).distinct().zipWithUniqueId()
</code></pre>

<p>After that, I did</p>

<pre><code>data = data.map(lambda x: Rating(int(user.lookup(x[0])), int(x[1]), float(x[2]))) 
</code></pre>

<p>What I ultimately want to do is run an ALS model on it, but so far I have been getting this error message </p>

<blockquote>
  <p>Exception: It appears that you are attempting to broadcast an RDD or reference an RDD from an action or transformation. </p>
</blockquote>

<p>I think the data type is somehow wrong, but I am not sure how to fix this. </p>