Pandas compare all DataFrame columns with eachother
<p>How can I compare all columns in a DataFrame with each other, removing columns that are 'less than' other columns according to an arbitrary comparison function (where the comparison function is transitive).</p>

<p>e.g. If I have the DataFrame</p>

<pre><code>  0 1 2 3 4 5
0 4 1 1 3 6 2
1 4 2 2 7 2 6
2 4 3 3 3 6 2
3 4 8 3 7 2 6
</code></pre>

<p>and my function is "column A is &lt; column B if A[i] &lt; B[i] for all rows i", the result would be</p>

<pre><code>  0 1 3 4
0 4 1 3 6
1 4 2 7 2
2 4 3 3 6
3 4 8 7 2
</code></pre>

<p>dropping column 2 as (4>1, 4>2, 4>3, 4>3) and column 5 as (3>2, 7>6, 3>2, 7>6).</p>

<p>My initial/obvious-but-slow approach is that this can be done in n^2 time with (pseudo code; I haven't done much Pandas programming before... answers with real code would be appreciated)</p>

<pre><code>for i in range(0, n):
  for j in range(0, n):
    if my_less_than_function(col(i), col(j)):
      # i &lt; j
      drop col(i)
</code></pre>

<p>If the less than function is transitive I could also remember which columns I've already dropped and skip them when iterating i and j. I could also iterate j in range(i + 1, n) if my comparison function returned (-1, 0, 1) for (less, equal, more) instead of (true, false) for (less, equal or more)</p>

<p>Note the comparison function may not be pairwise by row e.g. it could be sum(col A) &lt; sum(col B) or number_of_primes_in(col A) &lt; number_of_primes_in(col B)</p>

<p>Thank you</p>