Import data into MATLAB from SQL database: 2 mil rows takes 30 sec, but 6 mil takes 21 min?
<p><strong>I can import 2 million rows using Matlab's DB toolbox in about 36 seconds. How can I import 6 million rows in under twenty minutes?</strong></p>

<p>The final fetch step of the query below takes about 36 seconds.</p>

<pre><code>q = 'select ... from mytable limit 2000000'; %notice 2 mil limit
result = exec(conn, q);
final_result = fetch(result);  % Takes about 36 seconds.
</code></pre>

<p>My overall table has 6,097,227 rows. <strong>But if I do:</strong></p>

<pre><code>q = 'select ... from mytable';
result = exec(conn, q);
final_result = fetch(result);
</code></pre>

<p>MATLAB <em>entirely loses it</em> on the final fetch step! CPU usage goes to about 500-600% (i.e. 6/8 cores are being used), and it takes forever. Currently, with it set to fetch in 10k batches, it eventually finished in just over 21 minutes. </p>

<p><strong>Ideas? What to do?</strong> I really struggling to see how how this isn't at least roughly linear in the number of rows. Have I crossed some weird limit?</p>

<p>BTW: whole query and import into R takes about 43 seconds in R using PostgreSQL driver etc... and 0 fiddling around. I can import in Stata in a similar time using ODBC.  </p>

<p>Note: in the above queries <code>...</code> are 10 or so numeric variables: some integers, some double precision. None are text.</p>