Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
<p>I have a dataframe in Spark in which one of the columns contains an array.Now,I have written a separate UDF which converts the array to another array with distinct values in it only. See example below:</p>

<p>Ex: <strong>[24,23,27,23]</strong> should get converted to <strong>[24, 23, 27]</strong>
Code:</p>

<pre><code>def uniq_array(col_array):
    x = np.unique(col_array)
    return x
uniq_array_udf = udf(uniq_array,ArrayType(IntegerType()))

Df3 = Df2.withColumn("age_array_unique",uniq_array_udf(Df2.age_array))
</code></pre>

<p>In the above code, <code>Df2.age_array</code> is the array on which I am applying the UDF to get a different column <code>"age_array_unique"</code> which should contain only unique values in the array.</p>

<p>However, as soon as I run the command <code>Df3.show()</code>, I get the error:</p>

<blockquote>
  <p>net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)</p>
</blockquote>

<p>Can anyone please let me know why this is happening?</p>

<p>Thanks!</p>