How to redesign use of Elastic search Scroll API, because of memory limits
<p>I have a restriction on memory that my application uses.</p>

<p>I'm generating files consisting of data returned from Elastic Search.</p>

<p>As I need to get all the data that stored in ES I'm using <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html" rel="nofollow">Scroll API</a> to get data and that put it to some <code>Collection&lt;Foo&gt;</code>. For now ES contains about several thousands records of <code>Foo</code> that is returned.</p>

<p>So it is more like imperative approach, because i'm loading all data like:</p>

<pre><code>List&lt;Foo&gt; allFoos = FooSource.loadAllFoo(); 
</code></pre>

<p>And then do some processing, after which I'm saving results to different files.</p>

<p>Then this files are accessible from REST endpoint. </p>

<p>So what I'm looking for is some advice how to limit memory usage that this approach rises.</p>

<p>I was thinking of instead of putting all the data in memory and then operate on it do some sequential processing as the data comes available.</p>

<p>Do I need some Java Rx or Spring Batch technology in this case ? It helped me if you produce some code samples or even pseudo code will be alright.</p>