Joining PySpark DataFrames on nested field
<p>I want to perform a join between these two PySpark DataFrames:</p>



<pre class="lang-py prettyprint-override"><code>from pyspark import SparkContext
from pyspark.sql.functions import col

sc = SparkContext()

df1 = sc.parallelize([
    ['owner1', 'obj1', 0.5],
    ['owner1', 'obj1', 0.2],
    ['owner2', 'obj2', 0.1]
]).toDF(('owner', 'object', 'score'))

df2 = sc.parallelize(
          [Row(owner=u'owner1',
           objects=[Row(name=u'obj1', value=Row(fav=True, ratio=0.3))])]).toDF()
</code></pre>

<p>The join has to be performed on the name of the object, namely the field <em>name</em> inside <em>objects</em> for df2 and <em>object</em> for df1.</p>

<p>I am able to perform a SELECT on the nested field, as in</p>

<pre class="lang-py prettyprint-override"><code>df2.where(df2.owner == 'owner1').select(col("objects.value.ratio")).show()
</code></pre>

<p>but I am not able to run this join:</p>

<pre class="lang-py prettyprint-override"><code>df2.alias('u').join(df1.alias('s'), col('u.objects.name') == col('s.object'))
</code></pre>

<p>which returns error</p>

<blockquote>
  <p>pyspark.sql.utils.AnalysisException: u"cannot resolve
  '(objects.name = cast(object as double))' due to data type
  mismatch: differing types in '(objects.name = cast(object as
  double))' (array and double).;"</p>
</blockquote>

<p>Any ideas how to solve this?</p>