How to use the PySpark CountVectorizer on columns that maybe null
<p>I have a column in my Spark DataFrame:</p>

<pre><code> |-- topics_A: array (nullable = true)
 |    |-- element: string (containsNull = true)
</code></pre>

<p>I'm using CountVectorizer on it:</p>

<pre><code>topic_vectorizer_A = CountVectorizer(inputCol="topics_A", outputCol="topics_vec_A")
</code></pre>

<p>I get NullPointerExceptions, because sometimes the topic_A column contains null.</p>

<p>Is there a way around this? Filling it with a zero-length array would work ok (although it will blow out the data size quite a lot) - but I can't work out how to do a fillNa on an Array column in PySpark.</p>