Create a dataframe from a list in pyspark.sql
<p>I am totally lost in a wired situation. Now I have a list <code>li</code></p>
<pre><code>li = example_data.map(lambda x: get_labeled_prediction(w,x)).collect()
print li, type(li)
</code></pre>
<p>the output is like,</p>
<pre><code>[(0.0, 59.0), (0.0, 51.0), (0.0, 81.0), (0.0, 8.0), (0.0, 86.0), (0.0, 86.0), (0.0, 60.0), (0.0, 54.0), (0.0, 54.0), (0.0, 84.0)] &lt;type 'list'&gt;
</code></pre>
<p>When I try to create a dataframe from this list:</p>
<pre><code>m = sqlContext.createDataFrame(l, [&quot;prediction&quot;, &quot;label&quot;])
</code></pre>
<p>It threw the error message:</p>
<pre><code>TypeError                                 Traceback (most recent call last)
&lt;ipython-input-90-4a49f7f67700&gt; in &lt;module&gt;()
 56 l = example_data.map(lambda x: get_labeled_prediction(w,x)).collect()
 57 print l, type(l)
---&gt; 58 m = sqlContext.createDataFrame(l, [&quot;prediction&quot;, &quot;label&quot;])
 59 '''
 60 g = example_data.map(lambda x:gradient_summand(w, x)).sum()

/databricks/spark/python/pyspark/sql/context.py in createDataFrame(self, data, schema, samplingRatio)
423             rdd, schema = self._createFromRDD(data, schema, samplingRatio)
424         else:
--&gt; 425             rdd, schema = self._createFromLocal(data, schema)
426         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
427         jdf = self._ssql_ctx.applySchemaToPythonRDD(jrdd.rdd(), schema.json())

/databricks/spark/python/pyspark/sql/context.py in _createFromLocal(self, data, schema)
339 
340         if schema is None or isinstance(schema, (list, tuple)):
--&gt; 341             struct = self._inferSchemaFromList(data)
342             if isinstance(schema, (list, tuple)):
343                 for i, name in enumerate(schema):

/databricks/spark/python/pyspark/sql/context.py in _inferSchemaFromList(self, data)
239             warnings.warn(&quot;inferring schema from dict is deprecated,&quot;
240                           &quot;please use pyspark.sql.Row instead&quot;)
--&gt; 241         schema = reduce(_merge_type, map(_infer_schema, data))
242         if _has_nulltype(schema):
243             raise ValueError(&quot;Some of types cannot be determined after inferring&quot;)

/databricks/spark/python/pyspark/sql/types.py in _infer_schema(row)
831         raise TypeError(&quot;Can not infer schema for type: %s&quot; % type(row))
832 
--&gt; 833     fields = [StructField(k, _infer_type(v), True) for k, v in items]
834     return StructType(fields)
835 

/databricks/spark/python/pyspark/sql/types.py in _infer_type(obj)
808             return _infer_schema(obj)
809         except TypeError:
--&gt; 810             raise TypeError(&quot;not supported type: %s&quot; % type(obj))
811 
812 

TypeError: not supported type: &lt;type 'numpy.float64'&gt;
</code></pre>
<p>But when I hard code this list in line:</p>
<pre><code>tt = sqlContext.createDataFrame([(0.0, 59.0), (0.0, 51.0), (0.0, 81.0), (0.0, 8.0), (0.0, 86.0), (0.0, 86.0), (0.0, 60.0), (0.0, 54.0), (0.0, 54.0), (0.0, 84.0)], [&quot;prediction&quot;, &quot;label&quot;])
tt.collect()
</code></pre>
<p>It works well.</p>
<pre><code>[Row(prediction=0.0, label=59.0),
 Row(prediction=0.0, label=51.0),
 Row(prediction=0.0, label=81.0),
 Row(prediction=0.0, label=8.0),
 Row(prediction=0.0, label=86.0),
 Row(prediction=0.0, label=86.0),
 Row(prediction=0.0, label=60.0),
 Row(prediction=0.0, label=54.0),
 Row(prediction=0.0, label=54.0),
 Row(prediction=0.0, label=84.0)]
</code></pre>
<p>what caused this problem and how to fix it? Any hint will be appreciated.</p>