Python pandas - remove group based on collective NaN count
<p>I have a dataset based on different weather stations for several variables (Temperature, Pressure, etc.),</p>

<pre><code>stationID | Time | Temperature | Pressure |...
----------+------+-------------+----------+
123       |  1   |     30      |  1010.5  |
123       |  2   |     31      |  1009.0  |
202       |  1   |     24      |  NaN     |
202       |  2   |     24.3    |  NaN     |
202       |  3   |     NaN     |  1000.3  |
...
</code></pre>

<p>And I would like to remove 'stationID' groups, which have more than a certain number of NaNs (taking into account all variables in the count). </p>

<p>If I try, </p>

<pre><code>df.loc[df.groupby('station')['temperature'].filter(lambda x: len(x[pd.isnull(x)] ) &lt; 30).index]
</code></pre>

<p>it works, as shown here: <a href="https://stackoverflow.com/questions/38572079/python-pandas-remove-groups-based-on-nan-count-threshold">Python pandas - remove groups based on NaN count threshold</a></p>

<p>But the above example takes into account 'temperature' only. So, <strong>how can I take into account the collective sum of NaNs of the available variables?</strong> i.e.: I would like to remove a group, where the collective sum of NaNs in [variable1, variable2, variable3,...] is less than a threshold.</p>