PySpark: filtering a DataFrame by date field in range where date is string
<p>My dataframes contains one field which is a date and it appears in the string format, as example</p>

<pre><code>'2015-07-02T11:22:21.050Z'
</code></pre>

<p>I need to filter the DataFrame on the date to get only the records in the last week. 
So, I was trying a map approach where I transformed the string dates to datetime objects with strptime: </p>

<pre><code>def map_to_datetime(row):
     format_string = '%Y-%m-%dT%H:%M:%S.%fZ'
     row.date = datetime.strptime(row.date, format_string)

df = df.map(map_to_datetime)
</code></pre>

<p>and then I would apply a filter as </p>

<pre><code>df.filter(lambda row:
    row.date &gt;= (datetime.today() - timedelta(days=7)))
</code></pre>

<p>I manage to get the mapping working but the filter fails with </p>

<blockquote>
  <p>TypeError: condition should be string or Column</p>
</blockquote>

<p>Is there a way to use a filtering in a way that works or should I change the approach and how?</p>