Spark sql how to explode without losing null values
<p>I have a Dataframe that I am trying to flatten. As part of the process, I want to explode it, so if I have a column of arrays, each value of the array will be used to create a separate row. For instance, </p>

<pre class="lang-none prettyprint-override"><code>id | name | likes
_______________________________
1  | Luke | [baseball, soccer]
</code></pre>

<p>should become </p>

<pre class="lang-none prettyprint-override"><code>id | name | likes
_______________________________
1  | Luke | baseball
1  | Luke | soccer
</code></pre>

<p>This is my code</p>

<pre class="lang-java prettyprint-override"><code>private DataFrame explodeDataFrame(DataFrame df) {
    DataFrame resultDf = df;
    for (StructField field : df.schema().fields()) {
        if (field.dataType() instanceof ArrayType) {
            resultDf = resultDf.withColumn(field.name(), org.apache.spark.sql.functions.explode(resultDf.col(field.name())));
            resultDf.show();
        }
    }
    return resultDf;
}
</code></pre>

<p>The problem is that in my data, some of the array columns have nulls. In that case, the entire row is deleted. So this dataframe:</p>

<pre class="lang-none prettyprint-override"><code>id | name | likes
_______________________________
1  | Luke | [baseball, soccer]
2  | Lucy | null
</code></pre>

<p>becomes </p>

<pre class="lang-none prettyprint-override"><code>id | name | likes
_______________________________
1  | Luke | baseball
1  | Luke | soccer
</code></pre>

<p>instead of </p>

<pre class="lang-none prettyprint-override"><code>id | name | likes
_______________________________
1  | Luke | baseball
1  | Luke | soccer
2  | Lucy | null
</code></pre>

<p>How can I explode my arrays so that I don't lose the null rows? </p>

<p>I am using Spark 1.5.2 and Java 8</p>