Unicode equal comparison failed while testing Spark with pytest
<p>I am testing Spark function which create RDD using pytest. Function looks like this:</p>

<pre><code>def joinDMS(dms,df):
    return dms.join(df,(df.mac == dms.mac) &amp; (df.ch == dms.ch),"right_outer")
</code></pre>

<p>My test function looks like this:</p>

<pre><code>def test_joinDMS(spark_context, hive_context):
    input_rdd_data = [
        ["0004", 46]
    ]
    input_rdd = spark_context.parallelize(input_rdd_data)
    df_input = hive_context.createDataFrame(input_rdd, ['mac', 'ch'])

    input_dms_data = [
        ["0004", "gotv", 46]
    ]
    input_dms = spark_context.parallelize(input_dms_data)
    df_dms = hive_context.createDataFrame(input_dms, ['mac', 'tech', 'ch'])

    expected_results = [
        ["0004", "gotv", 46, "0004", 46]
    ]
    input_exp_results = spark_context.parallelize(expected_results)
    df_exp_results = hive_context.createDataFrame(input_exp_results, ['mac', 'tech', 'ch', 'mac', 'ch'])

    results = tv_functions.joinDMS(df_dms, df_input)
    assert results == df_exp_results
</code></pre>

<p>Dataframe <code>results</code> and <code>df_exp_results</code> are identical, but test status is 'FAILED'. This is the error:</p>

<pre><code>---------------------- Captured stderr call -----------------------
/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/worker.py:48: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/worker.py:48: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/worker.py:48: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
</code></pre>

<p>Does anyone knows what is the reason of this issue? I think it might be the problem with strings, but not sure. </p>