How to read a file using multiple threads in Java when a high throughput(3GB/s) file system is available
<p>I understand that for a normal Spindle Drive system, reading files using multiple threads is inefficient.</p>

<p><strong>This is a different case, I have a high-throughput file systems available to me, which provides read speeds up to 3GB/s, with 196 CPU cores and 2TB RAM</strong></p>

<p>A single threaded Java program reads the file with maximum 85-100 MB/s, so I have potential to get better than single thread. I have to read files as big as 1TB in size and I have enough RAM to load it.</p>

<p>Currently I use the following or something similar, but need to write something with multi-threading to get better throughput:</p>

<p>Java 7 Files: 50 MB/s</p>

<pre><code>List&lt;String&gt; lines = Files.readAllLines(Paths.get(path), encoding);
</code></pre>

<p>Java commons-io: 48 MB/s</p>

<pre><code>List&lt;String&gt; lines = FileUtils.readLines(new File("/path/to/file.txt"), "utf-8");
</code></pre>

<p>The same with guava: 45 MB/s</p>

<pre><code>List&lt;String&gt; lines = Files.readLines(new File("/path/to/file.txt"), Charset.forName("utf-8"));
</code></pre>

<p>Java Scanner Class: Very Slow</p>

<pre><code>Scanner s = new Scanner(new File("filepath"));
ArrayList&lt;String&gt; list = new ArrayList&lt;String&gt;();
while (s.hasNext()){
    list.add(s.next());
}
s.close();
</code></pre>

<p><em>I want to be able to load the file and build the same ArrayList, in the correct sorted sequence, as fast as possible.</em></p>

<p>There is <a href="https://stackoverflow.com/questions/26226819/java-i-o-unexpected-performance-difference-between-sequentially-and-concurrentl">another question</a> that reads similar, but it is actually different, because of :
The question is discussing about systems where multi-threaded file I/O is physically impossible to be efficient, but due to technological advancements, we now have systems that are designed to support high-throughput I/O , and so the limiting factor is CPU/SW , which can be overcome by multi-threading the I/O.</p>

<p>The other question does not answer how to write code to multi-thread I/O.</p>