PySpark dataframe: filter records with four or more non-null columns
<p>I have a number of PySpark dataframes where the data in two of the columns are mandatory, and the other columns are optional. The mandatory columns contain a date and a record ID; the most valuable data resides in the optional columns. I'm trying to capture the connections between elements in the optional columns.</p>

<p>The dataframe, pre-filter:</p>

<pre><code>id     col1    col2    col3    date
123            xyz             20160401
234    abc     pqr             20160401
345    def     hij     klm     20160401
456                            20160401
</code></pre>

<p>Post-filter, the dataframe would look like this:</p>

<pre><code>id     col1    col2    col3    date
234    abc     pqr             20160401
345    def     hij     klm     20160401
</code></pre>

<p>The records with multiple non-null column values are interesting because they describe relationships.</p>

<p>I notice that PySpark has a <code>.filter</code> method. The examples in the documentation typically show filtering a column, e.g. <code>df_filtered = df.filter(df.some_col &gt; some_value)</code>. I'm trying to write a filter to capture all those records with four or more non-null columns for an <em>arbitrary</em> dataframe, i.e. the column names must not be explicitly stated.</p>

<p>Is there an easy way to do this in PySpark?</p>

<h3>Update</h3>

<p>Although <code>.dropna(thresh=4)</code> seems to be exactly what I'm looking for, for some reason it didn't work. E.g.</p>

<pre><code>df.collect()

[Row(id=123, col1=None,         col2=None, col3=3754907743, date='20160403'),
 Row(id=124, col1=7911019393,   col2=None, col3=1456473867, date='20160403'),
 Row(id=125, col1=None,         col2=None, col3=2049622472, date='20160403'),
 Row(id=126, col1=4345043212,   col2=None, col3=3168577324, date='20160403'),
 Row(id=127, col1=None,         col2=None, col3=3185277065, date='20160403'),
 Row(id=128, col1=1336048242,   col2=None, col3=1322345860, date='20160403')]
</code></pre>

<p>Regardless of the <code>thresh</code> number, it always returns all the records in the original dataframe:</p>

<pre><code>df_filtered = df.dropna(thresh=[any number])

df_filtered.collect()

[Row(id=123, col1=None,         col2=None, col3=3754907743, date='20160403'),
 Row(id=124, col1=7911019393,   col2=None, col3=1456473867, date='20160403'),
 Row(id=125, col1=None,         col2=None, col3=2049622472, date='20160403'),
 Row(id=126, col1=4345043212,   col2=None, col3=3168577324, date='20160403'),
 Row(id=127, col1=None,         col2=None, col3=3185277065, date='20160403'),
 Row(id=128, col1=1336048242,   col2=None, col3=1322345860, date='20160403')]
</code></pre>

<p>I'm running Spark version 1.5.0-cdh5.5.2.</p>