Filter Pyspark dataframe column with None value
<p>I'm trying to filter a PySpark dataframe that has <code>None</code> as a row value:</p>

<pre><code>df.select('dt_mvmt').distinct().collect()

[Row(dt_mvmt=u'2016-03-27'),
 Row(dt_mvmt=u'2016-03-28'),
 Row(dt_mvmt=u'2016-03-29'),
 Row(dt_mvmt=None),
 Row(dt_mvmt=u'2016-03-30'),
 Row(dt_mvmt=u'2016-03-31')]
</code></pre>

<p>and I can filter correctly with an string value:</p>

<pre><code>df[df.dt_mvmt == '2016-03-31']
# some results here
</code></pre>

<p>but this fails:</p>

<pre><code>df[df.dt_mvmt == None].count()
0
df[df.dt_mvmt != None].count()
0
</code></pre>

<p>But there are definitely values on each category. What's going on?</p>