Create DataFrame from list of tuples using pyspark
<p>I am working with data extracted from SFDC using simple-salesforce package.
I am using Python3 for scripting and Spark 1.5.2.</p>

<p>I created an rdd containing the following data:</p>

<pre><code>[('Id', 'a0w1a0000003xB1A'), ('PackSize', 1.0), ('Name', 'A')]
[('Id', 'a0w1a0000003xAAI'), ('PackSize', 1.0), ('Name', 'B')]
[('Id', 'a0w1a00000xB3AAI'), ('PackSize', 30.0), ('Name', 'C')]
...
</code></pre>

<p>This data is in RDD called v_rdd</p>

<p>My schema looks like this:</p>

<pre><code>StructType(List(StructField(Id,StringType,true),StructField(PackSize,StringType,true),StructField(Name,StringType,true)))
</code></pre>

<p>I am trying to create DataFrame out of this RDD:</p>

<pre><code>sqlDataFrame = sqlContext.createDataFrame(v_rdd, schema)
</code></pre>

<p>I print my DataFrame:</p>

<pre><code>sqlDataFrame.printSchema()
</code></pre>

<p>And get the following:</p>

<pre><code>+--------------------+--------------------+--------------------+
|                  Id|  PackSize|                          Name|
+--------------------+--------------------+--------------------+
|[Ljava.lang.Objec...|[Ljava.lang.Objec...|[Ljava.lang.Objec...|
|[Ljava.lang.Objec...|[Ljava.lang.Objec...|[Ljava.lang.Objec...|
|[Ljava.lang.Objec...|[Ljava.lang.Objec...|[Ljava.lang.Objec...|
</code></pre>

<p>I am expecting to see actual data, like this:</p>

<pre><code>+------------------+------------------+--------------------+
|                Id|PackSize|                          Name|
+------------------+------------------+--------------------+
|a0w1a0000003xB1A  |               1.0|       A            |
|a0w1a0000003xAAI  |               1.0|       B            |
|a0w1a00000xB3AAI  |              30.0|       C            |
</code></pre>

<p>Can you please help me identify what I am doing wrong here.</p>

<p>My Python script is long, I am not sure it would be convenient for people to sift through it, so I posted only parts I am having issue with. </p>

<p>Thank a ton in advance!</p>