Pandas dataframe pivot not fitting in memory
<p>I have a dataframe <code>df</code> with the following structure:</p>

<pre><code>        val          newidx    Code
Idx                             
0       1.0      1220121127    706
1       1.0      1220121030    706
2       1.0      1620120122    565
</code></pre>

<p>It has 1000000 lines. 
In total we have 600 unique <code>Code</code> value and 200000 unique <code>newidx</code> values.</p>

<p>If I perform the following operation</p>

<pre><code>df.pivot_table(values='val', index='newidx', columns='Code', aggfunc='max')
</code></pre>

<p>I get a <code>MemoryError</code> . but this sounds strange as the size of the resulting dataframe should be sustainable: 200000x600.</p>

<p>How much memory requires such operation? Is there a way to fix this memory error?</p>