Spark dataframe after join null check for integer type columns
<p>I am implemeting solutions for removing duplicate elements from two dataframes using left out join. After performing join condition, I have to check null columns for right table. </p>

<pre><code>val llist = Seq(("bob", "2015-01-13", 4), ("alice", "2015-04-23",10))
val left = llist.toDF("name","date","duration")
val right = Seq(("alice", "2015-04-23",10),("bob", "2015-04-23", 23)).toDF("name","date","duration")


val df = left.join(right , left("name") === right("name") &amp;&amp;
  left("date") === right("date") &amp;&amp;
  left("duration").cast(StringType) === right("duration").cast(StringType)
  ,"left_outer").filter(right("duration").isNull)
</code></pre>

<p>But I am unable to filter out integer columns with null values. How can we do null check for integers after join ?</p>