Pyspark Enforce Nullability on Read of Json File
<p>I'm trying to read a json file and enforce a schema on read using SQLContext but the nullability part seems to get ignored. 
I have a schema something like this:</p>

<pre><code>StructType(List(StructField(some_field,StringType,false), StructField(some_other_field,StringType,false))
</code></pre>

<p>Now then, I want to read a json file and enforce that schema onto it as such:</p>

<pre><code>sqlc = SQLContext(sc)
df = sqlc.read.load("path/to/file", format="json", schema=schema)
</code></pre>

<p>The field names and data type seem to work just fine, but no matter what I put in for nullability, all the columns have nullable = true as such:</p>

<pre><code>root
 |--some_field: string (nullable = true)
 |--some_other_field: string (nullable = true)
</code></pre>

<p>How can I enforce nullability on my columns on read?</p>

<p>FYI. I'm using Python 2.7, pyspark 1.5.2</p>