Memory error when running medium sized merge function ipython notebook jupyter
<p>I'm trying to merge around 100 dataframes with a for loop and am getting a memory error. I'm using ipython jupyter notebook </p>

<p>Here is a sample of the data:</p>

<pre><code>    timestamp   Namecoin_cap
0   2013-04-28  5969081
1   2013-04-29  7006114
2   2013-04-30  7049003
</code></pre>

<p>Each frame is around 1000 lines long </p>

<p><a href="http://pastebin.com/QTcPQ1jY" rel="nofollow">Here's the error in detail, I've also include my merge function.</a> </p>

<p><a href="http://i.imgur.com/YqyEse0.png" rel="nofollow">My system is currently using up 64% of it memory</a>  </p>

<p>I have searched for similar issues but it seems most are for very large arrays >1GB, my data is relatively small in comparison.</p>

<p>EDIT: Something is suspicious. I wrote a beta program before, this was to test with 4 dataframes, i just exported that through pickle and it is 500kb. Now when i try to export the 100 frames one I get a memory error. It does however export a file that is 2GB. So i suspect somewhere down the line my code has created some kind of loop, creating a very large file. NB the 100 frames are stored in a dictionary</p>

<p>EDIT2: I have exported the scrypt to .py </p>

<p><a href="http://pastebin.com/GqaHr7xc" rel="nofollow">http://pastebin.com/GqaHr7xc</a></p>

<p><a href="https://www.mediafire.com/?tev25yznxk2b6cd" rel="nofollow">This is a .xlsx that cointains asset names the script needs</a> </p>

<p>The script fetches data regarding various assets, then cleans it up and saves each asset to a data frame in a dictionary</p>

<p>I'd be really appreciative if someone could have a look and see if there's anything immediately wrong. Other wise please advise on what tests I can run.</p>

<p>EDIT3: I'm finding it really hard to understand why this is happening, the code worked fine in the beta, all i have done now is add more assets. </p>

<p>EDIT4: I ran I size check on the object (dict of dfs) and it is 1,066,793 bytes</p>

<p>EDIT5: The problem is in the merge function for coin 37 </p>

<pre><code>for coin in coins[:37]:
    data2['merged'] = pd.merge(left=data2['merged'],right=data2[coin], left_on='timestamp', right_on='timestamp', how='left')
</code></pre>

<p>This is when the error occurs.  <code>for coin in coins[:36]:' doesn't produce an error however</code>for coin in coins[:37]:' produces the error, any ideas ? </p>

<p>EDIT6: the 36th element is 'Syscoin', i did coins.remove('Syscoin') however the memory problem still occurs. So it seems to be a problem with the 36th element in coins no matter what the coin is </p>

<p>EDIT7: goCards suggestions seemed to work however the next part of the code:</p>

<pre><code>merged = data2['merged']
merged['Total_MC'] = merged.drop('timestamp',axis=1).sum(axis=1)
</code></pre>

<p>Produces a memory error. I'm stumped </p>