Spark: Match columns from two dataframes
<p>I have a dataframe of format as below</p>

<pre><code>+---+---+------+---+
| sp|sp2|colour|sp3|
+---+---+------+---+
|  0|  1|     1|  0|
|  1|  0|     0|  1|
|  0|  0|     1|  0|
+---+---+------+---+
</code></pre>

<p>another dataframe contains coefficients for each column in first dataframe. for example</p>

<pre><code>+------+------+---------+------+
| CE_sp|CE_sp2|CE_colour|CE_sp3|
+------+------+---------+------+
|  0.94|  0.31|     0.11|  0.72|
+------+------+---------+------+
</code></pre>

<p>Now I want to add a column to first dataframe which is calculated by adding scores from second dataframe.</p>

<p>for ex.</p>

<pre><code>+---+---+------+---+-----+
| sp|sp2|colour|sp3|Score|
+---+---+------+---+-----+
|  0|  1|     1|  0| 0.42|
|  1|  0|     0|  1| 1.66|
|  0|  0|     1|  0| 0.11|
+---+---+------+---+-----+
</code></pre>

<p>i.e </p>

<pre><code>r -&gt; row of first dataframe
score = r(0)*CE_sp + r(1)*CE_sp2 + r(2)*CE_colour + r(3)*CE_sp3
</code></pre>

<p>There can be n number of columns and order of columns can be different.</p>

<p>Thanks in Advance!!!</p>