Pyspark Dataframe Apply function to two columns
<p>Say I have two PySpark DataFrames <code>df1</code> and <code>df2</code>. </p>

<pre><code>df1=   'a' 
        1    
        2    
        5    

df2=   'b'
        3
        6
</code></pre>

<p>And I want to find the closest <code>df2['b']</code> value for each <code>df1['a']</code>, and add the closest values as a new column in <code>df1</code>.</p>

<p>In other words, for each value <code>x</code> in <code>df1['a']</code>, I want to find a <code>y</code> that achieves <code>min(abx(x-y))</code> for all <code>y in df2['b']</code>(note: can assume that there is only one <code>y</code> that can achieve the minimum distance), and the result would be</p>

<pre><code>'a'    'b'
 1      3
 2      3
 5      6
</code></pre>

<p>I tried the following code to create a distance matrix first (before finding the values achieving the minimum distance):</p>

<pre><code>from pyspark.sql.types import IntegerType
from pyspark.sql.functions import udf

def dict(x,y):
    return abs(x-y)
udf_dict = udf(dict, IntegerType())

sql_sc = SQLContext(sc)
udf_dict(df1.a, df2.b)
</code></pre>

<p>which gives </p>

<pre><code>Column&lt;PythonUDF#dist(a,b)&gt;
</code></pre>

<p>Then I tried </p>

<pre><code>sql_sc.CreateDataFrame(udf_dict(df1.a, df2.b))
</code></pre>

<p>which runs forever without giving error/output.</p>

<p>My questions are:</p>

<ol>
<li>As I'm new to Spark, is my way to construct the output DataFrame efficient? (My way would be creating a distance matrix for all the <code>a</code> and <code>b</code> values first, and then find the <code>min</code> one)</li>
<li>What's wrong with the last line of my code and how to fix it?</li>
</ol>