Back-reference in Spark DataFrame `regexp_replace`
<p>I was recently trying to answer a <a href="https://stackoverflow.com/questions/40146760/splitting-row-in-multiple-row-in-spark-shell/40148915#40148915">question</a>, when I realised I didn't know how to use a back-reference in a regexp with Spark DataFrames.</p>

<p>For instance, with sed, I could do</p>

<pre><code>&gt; echo 'a1
b22
333' | sed "s/\([0-9][0-9]*\)/;\1/"                                                                                                   

a;1
b;22
;333
</code></pre>

<p>But with Spark DataFrames I can't:</p>

<pre><code>val df = List("a1","b22","333").toDF("str")
df.show

+---+
|str|
+---+
| a1|
|b22|
|333|
+---+

val res = df  .withColumn("repBackRef",regexp_replace('str,"(\\d+)$",";\\1"))
res.show

+---+-----------+
|str|repBackRef|
+---+----------+
| a1|       a;1|
|b22|       b;1|
|333|        ;1|
+---+----------+
</code></pre>

<p>Just to make it clear: I don't want the result in this particular case, I would like a solution that would be as generic as back reference in, for instance, <code>sed</code>.</p>

<p>Note also that using <code>regexp_extract</code> is lacking since it behaves badly when no matching:</p>

<pre><code>val res2 = df
  .withColumn("repExtract",regexp_extract('str,"^([A-z])+?(\\d+)$",2))
res2.show
</code></pre>

<p>So that you are forced to use one column per pattern to extract as I did in the said <a href="https://stackoverflow.com/questions/40146760/splitting-row-in-multiple-row-in-spark-shell/40148915#40148915">answer</a>.</p>

<p>Thanks!</p>