How to modify/transform the column of a dataframe?
<p>I have an instance of <code>pyspark.sql.dataframe.DataFrame</code> created using </p>

<pre><code>dataframe = sqlContext.sql("select * from table").
</code></pre>

<p>One column is 'arrival_date' and contains a string. </p>

<p>How do I modify this column so as to only take the first 4 characters from it and throw away the rest?</p>

<p>How would I convert the type of this column from string to date? </p>

<p>In graphlab.SFrame, this would be:</p>

<pre><code>dataframe['column_name'] = dataframe['column_name'].apply(lambda x: x[:4] )
</code></pre>

<p>and</p>

<pre><code>dataframe['column_name'] = dataframe['column_name'].str_to_datetime()
</code></pre>