what is the optimal chunksize in pandas read_csv to maximize speed?
<p>I am using a 20GB (compressed) .csv file and I load a couple of columns from it using pandas <code>pd.read_csv()</code> with a chunksize=10,000 parameter.</p>

<p>However, this parameter is completely arbitrary and I wonder whether a simple formula could give me better chunksize that would speed-up the loading of the data. </p>

<p>Any ideas?</p>