Lazy CSV Filtering / Parsing - Increasing Performance
<h1>Lazy Filtering CSV Files</h1>

<p>I had the need to filter through millions of log records, stored as numerous CSV files. The size of the records greatly exceeded my available memory so I wanted to go with a lazy approach.</p>

<h2>Java 8 Streams API</h2>

<p>With <code>jdk8</code> we have the Streams API which paired with Apache <code>commons-csv</code> allows us to easily accomplish this.</p>

<pre><code>public class LazyFilterer {

    private static Iterable&lt;CSVRecord&gt; getIterable(String fileName) throws IOException {
        return CSVFormat
                .DEFAULT
                .withFirstRecordAsHeader()
                .parse(new BufferedReader(new FileReader(fileName)));
    }

    public static void main(String[] args) throws Exception {
        File dir = new File("csv");

        for (File file : dir.listFiles()) {
            Iterable&lt;CSVRecord&gt; iterable = getIterable(file.getAbsolutePath());

            StreamSupport.stream(iterable.spliterator(), true)
                    .filter(c -&gt; c.get("API_Call").equals("Updates"))
                    .filter(c -&gt; c.get("Remove").isEmpty())
                    .forEach(System.out::println);
        }
    }
}
</code></pre>

<h2>Performance</h2>

<p>This graph from VisualVM shows the memory usage during the parsing of 2.3 GB of CSV files using a more complex filtration pipeline<sup>1</sup> than shown above. </p>

<p>As you can see, the memory usage basically remains constant<sup>2</sup> as the filtration occurs.</p>

<p><img src="https://i.imgur.com/f2Mvn5a.png" alt="visualvm screenshot"></p>

<p><strong>Can you find another method to accomplish the same task more quickly while not increasing code complexity?</strong></p>

<p>Any languages are welcome, Java is not necessarily preferred!</p>

<h1>Footnotes</h1>

<p><sub>
[1] - E.g. for each <code>CSVRecord</code> that matches on <code>"API_Call"</code> I might need to do some JSON deserialization and do additional filtering after that, or even create an object for certain records to facilitate additional computations.</sub></p>

<p><sub>
[2] - The idle time at the beginning of the graph was a <code>System.in.read()</code> used to ensure that VisualVM was fully loaded before computation began.</sub></p>