Converting Pandas DataFrame to Spark DataFrame
<p>I had asked a previous question about how to <a href="https://stackoverflow.com/questions/40353018/convert-scipy-sparse-matrix-to-pyspark-sql-dataframe-dataframe/40364070?noredirect=1#comment68034638_40364070">Convert scipy sparse matrix to pyspark.sql.dataframe.DataFrame</a>, and made some progress after reading the answer provided, as well as <a href="https://stackoverflow.com/questions/17818783/populate-a-pandas-sparsedataframe-from-a-scipy-sparse-matrix">this article</a>. I eventually came to the following code for converting a scipy.sparse.csc_matrix to a pandas dataframe:</p>

<pre><code>df = pd.DataFrame(csc_mat.todense()).to_sparse(fill_value=0)
df.columns = header
</code></pre>

<p>I then tried converting the pandas dataframe to a spark dataframe using the suggested syntax:</p>

<pre><code>spark_df = sqlContext.createDataFrame(df)
</code></pre>

<p>However, I get back the following error:</p>

<pre><code>ValueError: cannot create an RDD from type: &lt;type 'list'&gt;
</code></pre>

<p>I do not believe it has anything to do with the sqlContext as I was able to convert another pandas dataframe of roughly the same size to a spark dataframe, no problem. Any thoughts?</p>