spark full outer join issue
<p>My main use case is to perform code migration from SAS to Spark-scala.This may be a bug or possibly I might be doing something wrong, but while using Full outer join,or even with A left outer join B union all B right outer join A  and exclusion join(where a.col1 is null) combination, (as alternative of full outer join condition) in my code, it is returning different no of rows on each occasion. The input datasets have same number of rows in SAS and Spark, and the output always has more rows than the SAS counterpart. </p>

<p>This my sample code for example:</p>

<pre><code>data1.registerTempTable("tblA")
data2.registerTempTable("tblB")

val query = """SELECT A.*,B.* FROM tblA A FULL OUTER JOIN  tblB  B
ON UCASE(trim(A.col1))=UCASE(trim(B.col1)) AND trim(A.col2)=trim(B.col2) AND
a.col3=LPAD(B.col3,6,'0') """

val result = sqlContext.sql(query)

result.count
</code></pre>

<p>Additional info: I even tried with</p>

<p>Step1) A left outer join B union all A right outer join B,and then 
Step2) using .dropDuplicates() on the resulting dataframe, </p>

<p>Step 1 gives the same result on multiple iterations, while combination of step 1 and 2 again gives different results.</p>

<p>Anyone have any ideas?</p>