How to use a function over an RDD and get new column (Pyspark)?
<p>I'm looking for a way to apply a function to an RDD using PySpark and put the result in a new column. With DataFrames, it looks easy :
Given :</p>

<pre><code>rdd = sc.parallelize([(u'1751940903', u'2014-06-19', '2016-10-19'), (u'_guid_VubEgxvPPSIb7W5caP-lXg==', u'2014-09-10', '2016-10-19')])
</code></pre>

<p>My code can look like this :</p>

<pre><code>df= rdd.toDF(['gigya', 'inscription','d_date'])
df.show()
+--------------------+-------------------------+----------+
|               gigya|              inscription|    d_date|
+--------------------+-------------------------+----------+
|          1751940903|               2014-06-19|2016-10-19|
|_guid_VubEgxvPPSI...|               2014-09-10|2016-10-19|
+--------------------+-------------------------+----------+
</code></pre>

<p>Then :</p>

<pre><code>from pyspark.sql.functions import split, udf, col
get_period_day = udf(lambda item : datetime.strptime(item, "%Y-%m-%d").timetuple().tm_yday)

df.select('d_date', 'gigya', 'inscription', get_period_day(col('d_date')).alias('period_day')).show()

+----------+--------------------+-------------------------+----------+
|    d_date|               gigya|inscription_service_6Play|period_day|
+----------+--------------------+-------------------------+----------+
|2016-10-19|          1751940903|               2014-06-19|       293|
|2016-10-19|_guid_VubEgxvPPSI...|               2014-09-10|       293|
+----------+--------------------+-------------------------+----------+
</code></pre>

<p>Is there a way to do the same thing without the need to convert my RDD to a DataFrame ? Something with map for exemple..</p>

<p>This code can just give me a part from the expected results :</p>

<pre><code>rdd.map(lambda x: datetime.strptime(x[1], '%Y-%m-%d').timetuple().tm_yday).cache().collect()
</code></pre>

<p>Help ?</p>