Applying a Window function to calculate differences in pySpark
<p>
I am using <code>pySpark</code>, and have set up my dataframe with two columns representing a daily asset price as follows:</p>

<pre class="lang-python prettyprint-override"><code>ind = sc.parallelize(range(1,5))
prices = sc.parallelize([33.3,31.1,51.2,21.3])
data = ind.zip(prices)
df = sqlCtx.createDataFrame(data,["day","price"])
</code></pre>

<p>I get upon applying <code>df.show()</code>:</p>

<pre class="lang-python prettyprint-override"><code>+---+-----+
|day|price|
+---+-----+
|  1| 33.3|
|  2| 31.1|
|  3| 51.2|
|  4| 21.3|
+---+-----+
</code></pre>

<p>Which is fine and all. I would like to have another column that contains the day-to-day returns of the price column, i.e., something like </p>

<p><code>(price(day2)-price(day1))/(price(day1))</code></p>

<p>After much research, I am told that this is most efficiently accomplished through applying the <code>pyspark.sql.window</code> functions, but I am unable to see how.</p>