How to join two data frames in Apache Spark and merge keys into one column?
<p>I have two following Spark data frames:</p>

<pre><code>sale_df:

|user_id|total_sale|
+-------+----------+
|      a|      1100|
|      b|      2100|
|      c|      3300|
|      d|      4400  
</code></pre>

<p>and target_df:</p>

<pre><code> user_id|personalized_target|
+-------+-------------------+
|      b|               1000|
|      c|               2000|
|      d|               3000|
|      e|               4000|
+-------+-------------------+
</code></pre>

<p>How can I join them in a way that output is:</p>

<pre><code>user_id   total_sale   personalized_target
 a           1100            NA
 b           2100            1000
 c           3300            2000
 d           4400            4000
 e           NA              4000
</code></pre>

<p>I have tried all most all the join types but it seems that single join can not make the desired output.</p>

<p>Any PySpark or SQL and HiveContext can help.</p>