Calculate quantile on grouped data in spark Dataframe
<p>I have the following Spark dataframe :</p>

<pre><code> agent_id|payment_amount|
+--------+--------------+
|       a|          1000|
|       b|          1100|
|       a|          1100|
|       a|          1200|
|       b|          1200|
|       b|          1250|
|       a|         10000|
|       b|          9000|
+--------+--------------+
</code></pre>

<p>my desire output would be something like </p>

<pre><code>agen_id   95_quantile
  a          whatever is 95 quantile for agent a payments
  b          whatever is 95 quantile for agent b payments
</code></pre>

<p>for each group of agent_id I need to calculate the 0.95 quantile, I take the following approach:</p>

<pre><code>test_df.groupby('agent_id').approxQuantile('payment_amount',0.95)
</code></pre>

<p>but I take the following error:</p>

<pre><code>'GroupedData' object has no attribute 'approxQuantile'
</code></pre>

<p>I need to have .95 quantile(percentile) in a new column so later can be used for filtering purposes</p>

<p>I am using Spark 2.0.0</p>