Catalyst optimizer error when having a loop in Spark
<p>I have a spark program in which I'm using a loop and generate some Dataframes within the loop. It sometimes work fine, however it gives a Catalyst error sometimes. Here is the error message.</p>

<pre><code>org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
TungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[count#271L])
+- TungstenExchange SinglePartition, None
+- TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#274L])
  +- Project
     +- SortMergeJoin [cust_id#204L], [cust_id#234L]
        :- Project [cust_id#204L]
        :  +- SortMergeJoin [cust_id#204L], [cust_id#228L]
        :     :- Project [cust_id#204L]
        :     :  +- SortMergeJoin [cust_id#204L], [cust_id#222L]
        :     :     :- Project [cust_id#204L]
        :     :     :  +- SortMergeJoin [cust_id#204L], [cust_id#216L]
        :     :     :     :- Project [cust_id#204L]
        :     :     :     :  +- SortMergeJoin [cust_id#204L], [cust_id#210L]
        :     :     :     :     :- Sort [cust_id#204L ASC], false, 0
        :     :     :     :     :  +- TungstenExchange hashpartitioning(cust_id#204L,200), None
        :     :     :     :     :     +- Union
</code></pre>

<p>The code within the loop works fine, if I run it without the loop it works perfectly. Why I'm getting this error?</p>