"Task not serializable" error in pySpark on unionAll
<p>I'm trying to stack two small dataframes of summary statistics, each with the same columns in the same order and matching dtypes.  The first dataframe is simply the results of calling <code>describe</code> on (a subset of) my main dataframe; the second is two more types of summary statistics I assembled and want to append to the first.  When I try <code>df1.unionAll(df2)</code> I get the following error:</p>

<blockquote>
  <p>Py4JJavaError: An error occurred while calling o330.collectToPython. :
  org.apache.spark.SparkException: Task not serializable</p>
</blockquote>

<p>However, it works just fine if I collect them into lists, concatenate them, then parallelize them again:</p>

<pre><code>df1_c = df1.collect()
df1_c
</code></pre>

<blockquote>
  <p>[Row(summary=u'count', C0=u'3526154', C3=u'3526154', C4=u'1580402',
  C5=u'3526154', C6=u'3526154'),  Row(summary=u'mean',
  C0=u'5.50388599500189E11', C3=u'4.178168090221902',
  C4=u'234846.78065481802', C5=u'5.134865351881966',
  C6=u'354.7084951479714'),  Row(summary=u'stddev',
  C0=u'2.5961123619752225E11', C3=u'0.34382335723646484',
  C4=u'118170.68592261615', C5=u'3.3833930336063456',
  C6=u'4.011812510792076'),  Row(summary=u'min', C0=u'100002091588',
  C3=u'2.75', C4=u'0.85', C5=u'-1', C6=u'292'),  Row(summary=u'max',
  C0=u'999995696635', C3=u'6.125', C4=u'1193544.39', C5=u'34',
  C6=u'480')]</p>
</blockquote>

<pre><code>df2_c = df1.collect()
df2_c
</code></pre>

<blockquote>
  <p>[Row(summary=u'skew', C0=u'-0.00183847089857', C3=u'0.519799339496',
  C4=u'0.758411576756', C5=u'0.286480156084', C6=u'-2.69765201567'),
       Row(summary=u'kurtosis', C0=u'-1.19900726351', C3=u'0.126057726847', C4=u'0.576085602656', C5=u'0.195187780089',
  C6=u'24.7237858944')]</p>
</blockquote>

<pre><code>dd1 = df1.collect()
dd2 = df2.collect()
dd3 = dd1 + dd2
new_df = sc.parallelize(dd3).toDF()
new_df.show()
</code></pre>

<blockquote>
  <p>(A new dataframe with the previous two stacked just the way they
  should be, but looks messy here because it's too big so I'm leaving it off)</p>
</blockquote>

<p>I can also parallelize and call <code>toDF</code> on each collected list, then call <code>unionAll</code> on the new dataframes, <em>and that works fine.</em>  This operation should only ever happen on small dataframes because it's just summary statistics, but having to collect them into a list first obviously doesn't seem right.  And I've used <code>unionAll</code> successfully on dataframes in the past without issue.  Is there something different about a dataframe created from <code>describe</code> and another one created from parellizing <code>Row</code> objects?  They look identical using <code>show</code>, and when collected - the <code>collect</code> output above is actual data I'm having the problem with.  Any insight into why <code>unionAll</code> is upset here would be great.</p>

<p>Edit: To my dismay, I've been unable to make this a minimal reproducible example with toy data - that is, if I paralellize some random values, describe them, then create a row and try to stack it on the describe, it works fine.  This suggests there's something specific in my data or methodology that is throwing this off somehow.  Obviously this makes the prospects of finding a solution here a lot slimmer, so I apologize in advance.</p>