Difference between two DataFrames columns in pyspark
<p>I am looking for a way to find difference in values, in columns of two DataFrame. For example:</p>

<pre class="lang-python prettyprint-override"><code>from pyspark.sql import SQLContext

sc = SparkContext()
sql_context = SQLContext(sc)

df_a = sql_context.createDataFrame([("a", 3), ("b", 5), ("c", 7)], ["name", "id"])

df_b = sql_context.createDataFrame([("a", 3), ("b", 10), ("c", 13)], ["name", "id"])
</code></pre>

<p>DataFrame A:</p>

<pre class="lang-python prettyprint-override"><code>+----+---+
|name| id|
+----+---+
|   a|  3|
|   b|  5|
|   c|  7|
+----+---+
</code></pre>

<p>DataFrame B:</p>

<pre class="lang-python prettyprint-override"><code>+----+---+
|name| id|
+----+---+
|   a|  3|
|   b| 10|
|   c| 13|
+----+---+
</code></pre>

<p>My goal is a <code>list</code> of <code>id</code> column elements that are in A but not in B, e.g: <code>[5, 7]</code>. I was thinking of doing a join on <code>id</code>, but I don't see a good way to do it.</p>

<p>Naive solution could be:</p>

<pre class="lang-python prettyprint-override"><code>list_a = df_a.select("id").rdd.map(lambda x: x.asDict()["id"]).collect()
list_b = df_b.select("id").rdd.map(lambda x: x.asDict()["id"]).collect()

result = list(set(list_a).difference(list_b))
</code></pre>

<p>But, is there a simple solution that can be obtained with just DataFrame operations, except perhaps the final collect?</p>