Process a LOT of data
<p>So I'm working with parametric energy simulations and ended up with 500GB+ of data stored in <code>.CSV</code> files. I need to be able to process all this data to compare the results and gain insights of the influence of different parameters. </p>

<p>Each csv file name contains information of the parameters used for the simulation so I can not merge the files. </p>

<p>I normally loaded the .csv files to python using pandas and defining a Class. but now (with all this data) there is not enough memory to do this. </p>

<p>Can you point me out a way to process this data? I need to be able to do plots and compare the csv files. </p>

<p>Thank you for your time. </p>