Split Spark Dataframe string column into multiple columns
<p>I've seen various people suggesting that <code>Dataframe.explode</code> is a useful way to do this, but it results in more rows than the original dataframe, which isn't what I want at all. I simply want to do the Dataframe equivalent of the very simple:</p>

<pre><code>rdd.map(lambda row: row + [row.my_str_col.split('-')])
</code></pre>

<p>which takes something looking like:</p>

<pre><code>col1 | my_str_col
-----+-----------
  18 |  856-yygrm
 201 |  777-psgdg
</code></pre>

<p>and converts it to this:</p>

<pre><code>col1 | my_str_col | _col3 | _col4
-----+------------+-------+------
  18 |  856-yygrm |   856 | yygrm
 201 |  777-psgdg |   777 | psgdg
</code></pre>

<p>I am aware of <code>pyspark.sql.functions.split()</code>, but it results in a nested array column instead of two top-level columns like I want. </p>

<p>Ideally, I want these new columns to be named as well.</p>