How to release a dataframe in spark?
<p>I am using spark do some calculation. every 5 minutes, I got a new data frame. I put it into a dict called dict_1_hour like this</p>

<pre><code>dict_1_hour[timestamp] = dataframe
</code></pre>

<p>New data frame comes in to the dict and old dataframe pop out from the dict. Only 12 dataframe are kept in it, that is the data for the recent 1 hour.</p>

<p>So my question is how should I release those dataframe to make sure there is no memory leak?</p>

<p>One API for dataframe seems can do it.(I do not know what the parameter for)</p>

<pre><code>unpersist(blocking=True)
Marks the DataFrame as non-persistent, and remove all blocks for it from memory and disk.
</code></pre>

<p>The other way I think is just pop the dataframe from the dict.</p>

<pre><code>dict_1_hour.pop(timestamp)
</code></pre>

<p>Python should auto release the unused variable. But I do not know if it is appropriate here. I worried that spark may keep the dataframe if you do not release it explicit</p>

<p>So please advice on which way I should use please</p>