Pandas Dataframe: Lack of Memory- What's the better way here?
<pre><code>f = pd.read_hdf('Sensor_Data.h5','f')
pieces = [f[x: x + 360] for x in xrange(504649)]
df = pd.concat(pieces)
</code></pre>

<p>Morning all. I have a file with 500,000+ rows of data. I want to take 360 row slices from this, and move it down by 1 row each time. (So I will end up with a LOT of data. )</p>

<p>As expected, I tried the above code and got a memory error. I'm assuming there's a better way of doing this?</p>

<p>EDIT: To add some context, this is a .h5 file, and I'm using pandas dataframe to try and slice it this way. I'm trying to create an array of data to feed into a deep neural network using caffenet, though the format it will be in at this point will be unclear...</p>

<p>The code works for small amounts of data. Just not for larger ones. To be clearer of what I'm trying to do:import pandas as pd</p>

<pre><code>df = pd.DataFrame(np.random.randn(10,6)); df
</code></pre>

<p>[displays a 6 x 10 table of random numbers]
Now:</p>

<pre><code>pieces = [df[x: x + 4] for x in xrange(7)]
f = pd.concat(pieces)
f
</code></pre>

<p>Diplays a new table similar to the previous one, but expanded. It now has rows 0,1,2,3,1,2,3,4,2,3,4,5,3,4,5,6... </p>

<p>Now "pieces" is not a dataframe object itself, but a 'list' for some reason. Is there also a simple way to turn all of these separate datasets (0,1,2,3),(1,2,3,4) and so on, into dataframe object themselves? (Instead of concatenating them together into one dataframe?</p>

<p>I hope this makes sense.</p>