Easiest way to read csv files with multiprocessing in Pandas
<p>Here is my question.<br />
With bunch of .csv files(or other files). Pandas is an easy way to read them and save into <code>Dataframe</code> format. But when the amount of files was huge, I want to read the files with multiprocessing to save some time.</p>
<h3>My early attempt</h3>
<p>I manually divide the files into different path. Using severally:</p>
<pre class="lang-py prettyprint-override"><code>os.chdir(&quot;./task_1&quot;)
files = os.listdir('.')
files.sort()
for file in files:
    filename,extname = os.path.splitext(file)
    if extname == '.csv':
        f = pd.read_csv(file)
        df = (f.VALUE.as_matrix()).reshape(75,90)   
</code></pre>
<p>And then combine them.</p>
<p>How to run them with <code>pool</code> to achieve my problem?<br />
Any advice would be appreciated!</p>