Spark Scala: Issue Substituting Filter Expression In DataFrame
<p>I have a dataframe created which holds the join of 2 tables.</p>

<p>I want to compare each field of table1 to that of table2 (Schema is same)
 Columns in Table A = colA1, colB1, colC1 , ...
 Columns in Table B = colA2, colB2, colC2, ...
 So, I need to filter out the data which satisfies the condition</p>

<p>(colA1 = colA2) AND (colB1 = colB2) AND (colC1 = colC2) and so on.
 Since my table has a lot of fields, I tried to build a similar exp.</p>

<pre><code> val filterCols = Seq("colA","colB","colC")
 val sq = '"'
 val exp = filterCols.map({ x =&gt; s"(join_df1($sq${x}1$sq) === join_df1($sq${x}2$sq))" }).mkString(" &amp;&amp; ")
 Resultant Exp : res28: String = (join_df1("colA1") === join_df1("colA2")) &amp;&amp; (join_df1("colB1") === join_df1("colB2")) &amp;&amp; (join_df1("colC1") === join_df1("colC2"))
</code></pre>

<p>Now when i try to substitute it to the dataframe, it throws me an error.
 join_df1.filter($exp)</p>

<p>I am not sure whether I am doing it right .I need to find a way to substitute my expression and filter out value.
 Any help is appreciated. </p>

<p><a href="https://i.stack.imgur.com/i9gWW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/i9gWW.png" alt="enter image description here"></a>Thanks in advance</p>