pyspark.sql data.frame understanding functions
<p>I am taking a <a href="https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x" rel="nofollow noreferrer">mooc</a>.</p>

<p>It has one assignment where a column needs to be converted to the lower case. <code>sentence=lower(column)</code> does the trick. But initially I thought that the syntax should be <code>sentence=column.lower()</code>. I looked at the documentation and I couldnt figure out the problem with my syntax. Would it be possible to explain how I could have figured out that I have a wrong syntax by searching online documentation and function definition?</p>

<p>I am specially confused as <a href="https://stackoverflow.com/questions/6797984/how-to-convert-string-to-lowercase-in-python">This link</a> shows that string.lower() does the trick in case of the regular string python objects</p>

<pre><code>from pyspark.sql.functions import regexp_replace, trim, col, lower
def removePunctuation(column):
    """Removes punctuation, changes to lower case, and strips leading and trailing spaces.

    Note:
        Only spaces, letters, and numbers should be retained.  Other characters should should be
        eliminated (e.g. it's becomes its).  Leading and trailing spaces should be removed after
        punctuation is removed.

    Args:
        column (Column): A Column containing a sentence.

    Returns:
        Column: A Column named 'sentence' with clean-up operations applied.
    """

    sentence=lower(column)

    return sentence

sentenceDF = sqlContext.createDataFrame([('Hi, you!',),
                                         (' No under_score!',),
                                         (' *      Remove punctuation then spaces  * ',)], ['sentence'])
sentenceDF.show(truncate=False)
(sentenceDF
.select(removePunctuation(col('sentence')))
.show(truncate=False))
</code></pre>