PySpark DataFrame filter using logical AND over list of conditions -- Numpy All Equivalent
<p>I'm trying to filter rows of a PySpark dataframe if the values of all columns are zero.</p>
<p>I was hoping to use something like this, (using the numpy function <code>np.all()</code> ):</p>
<pre class="lang-py prettyprint-override"><code>from pyspark.sql.functions import col
df.filter(all([(col(c) != 0) for c in df.columns]))
</code></pre>
<p>But I get the ValueError:</p>
<pre><code>ValueError: Cannot convert column into bool: please use '&amp;' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
</code></pre>
<p>Is there any way to perform the logical and on a list of conditions? What is the corresponding <code>np.all</code> functionality in PySpark?</p>