Apply regex to every row of a spark dataframe and save it as a new column in the same dataframe
<p>Suppose I have a spark dataframe,</p>

<p>data.show()</p>

<pre><code>ID                URL
1             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed
2             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed
3             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed
4             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed
5                 None
</code></pre>

<p>I want to write a regex operation to it, where I want to parse the URL for a particular scenario. The scenario would be would be to parse out after &amp;q and before next &amp;. I am able to write this in python for a python dataframe as follows,</p>

<pre><code>re.sub(r"\s+", " ", re.search(r'/?q=([^&amp;]*)', data['url'][i]).group(1).replace('+', ' ')
</code></pre>

<p>I want to write the same in pyspark. </p>

<p>If a write something like,</p>

<pre><code> re.sub(r"\s+", " ", re.search(r'/?q=([^&amp;]*)', data.select(data.url.alias("url")).collect()).group(1).replace('+', ' '))
</code></pre>

<p>or</p>

<pre><code>re.sub(r"\s+", " ", re.search(r'/?q=([^&amp;]*)', data.select(data['url']).collect()).group(1).replace('+', ' '))
</code></pre>

<p>I am getting the following error,</p>

<pre><code>TypeError: expected string or buffer
</code></pre>

<p>One option is that to convert the data to pandas using,</p>

<p><code>data.toPandas()</code> and then do the operations. But my data is huge and converting it to pandas makes it slow. Is there a way I can write this directly to a new column in spark dataframe where I can have like,</p>

<pre><code>ID                URL                                                                           word
1             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed       To Be Parsed out   
2             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed       To Be Parsed out 
3             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed       To Be Parsed out 
4             https://www.sitename.com/&amp;q=To+Be+Parsed+out&amp;oq=Dont+Need+to+be+parsed       To Be Parsed out 
5             None                                                                             None
</code></pre>

<p>How can we do this to add it as a new table in pyspark dataframe? which applies to every row of the dataframe?</p>