Is dataframe.show() an action in spark?
<p>I have the following code:</p>

<pre><code>val df_in = sqlcontext.read.json(jsonFile) // the file resides in hdfs

//some operations in here to create df as df_in with two more columns "terms1" and "terms2" 

val intersectUDF = udf( (seq1:Seq[String], seq2:Seq[String] ) =&gt; {     seq1 intersect seq2 } ) //intersects two sequences
val symmDiffUDF = udf( (seq1:Seq[String], seq2:Seq[String] ) =&gt; { (seq1 diff seq2) ++ (seq2 diff seq1) } ) //compute the difference of two sequences

val df1 = (df.withColumn("termsInt", intersectUDF(df("terms1"), df1("terms2") ) )
             .withColumn("termsDiff", symmDiffUDF(df("terms1"),     df1("terms2") ) )
             .where( size(col("termsInt")) &gt;0 &amp;&amp; size(col("termsDiff")) &gt; 0 &amp;&amp; size(col("termsDiff")) &lt;= 2 )
             .cache()
           ) // add the intersection and difference columns and filter the resulting DF 

df1.show()
df1.count()
</code></pre>

<p>The app is working properly and fast until the <code>show()</code> but in the <code>count()</code> step, it creates 40000 tasks.</p>

<p>My understanding is that <code>df1.show()</code> should be triggering the full <code>df1</code> creation and  <code>df1.count()</code> should be very fast. What am I missing here? why is <code>count()</code> that slow?</p>

<p>Thank you  very much in advance,
Roxana</p>