converting pyspark DataFrame to LabeledPoint without dropping to RDD
<p>I am new to pyspark DataFrame, and this really bugs me somehow. Maybe I'm not understanding this correctly.  </p>

<p>say that I have a data frame</p>

<pre><code>a = sqlContext.createDataFrame([[(2,3,4)],[(1,2,3)]],['things'])
</code></pre>

<p>If I want to convert this to LabeledPoint, I need to drop down to RDD using map function. </p>

<pre><code>from pyspark.mllib.regression import LabeledPoint
def convert(x):
    z = [float(y) for y in x]
    return LabeledPoint(z[0], z[1:])

rdd = a.map(lambda x: convert(x['things']))
rdd.take(2)
df=rdd.toDF()
</code></pre>

<p>Why can't I use DataFrame select statement combined with UDF like below?</p>

<pre><code>from pyspark.sql.functions import udf
def convert(x):
   z = [float(y) for y in x]
   return LabeledPoint(z[0], z[1:])

udf_convert = udf(convert)
df = a.select(udf_convert(a['things']))
display(df)
</code></pre>

<p>I'm getting an error that says:</p>

<pre><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 33.0 failed 1 times, most recent failure: Lost task 6.0 in stage 33.0 (TID 101, localhost): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for pyspark.mllib.linalg.DenseVector)
</code></pre>