withColumn not allowing me to use max() function to generate a new column
<p>I have a dataset like this:</p>

<pre><code>a = sc.parallelize([[1,2,3],[0,2,1],[9,8,7]]).toDF(["one", "two", "three"])
</code></pre>

<p>I want to have a dataset that adds a new column that is equal to the largest value in the other three columns.
The output would look like this:</p>

<pre><code>+----+----+-----+-------+
|one |two |three|max_col|
+----+----+-----+-------+
|   1|   2|    3|      3|
|   0|   2|    1|      2|
|   9|   8|    7|      9|
+----+----+-----+-------+
</code></pre>

<p>I thought I would use <code>withColumn</code>, like so:</p>

<pre><code>b = a.withColumn("max_col", max(a["one"], a["two"], a["three"]))
</code></pre>

<p>but this yields the error </p>

<pre><code>Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/opt/spark152/python/pyspark/sql/column.py", line 418, in __nonzero__
    raise ValueError("Cannot convert column into bool: please use '&amp;' for 'and', '|' for 'or', "
ValueError: Cannot convert column into bool: please use '&amp;' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
</code></pre>

<p>Odd. Does <code>max</code> return a <code>bool</code>? Not according to <a href="https://docs.python.org/2/library/functions.html#max" rel="noreferrer">the documentation on <code>max</code></a>. Okay. Weird.</p>

<p>I find it odd that this works:</p>

<pre><code>b = a.withColumn("max_col", a["one"] + a["two"] + a["three"]))
</code></pre>

<p>And the fact that it works makes me think even more strongly that <code>max</code> is behaving some way I don't understand.</p>

<p>I also tried <code>b = a.withColumn("max_col", max([a["one"], a["two"], a["three"]]))</code>, which passes in the three columns as a list rather than 3 separte elements. This yields the same error as above.</p>