Scala - Batched Stream from Futures
<p>I have instances of a case class <code>Thing</code>, and I have a bunch of queries to run that return a collection of <code>Thing</code>s like so:</p>

<pre><code>def queries: Seq[Future[Seq[Thing]]]
</code></pre>

<p>I need to collect all <code>Thing</code>s from all futures (like above) and group them into equally sized collections of 10,000 so they can be serialized to files of 10,000 <code>Thing</code>s.</p>

<pre><code>def serializeThings(Seq[Thing]): Future[Unit]
</code></pre>

<p>I want it to be implemented in such a way that I don't wait for all queries to run before serializing. As soon as there are 10,000 <code>Thing</code>s returned after the futures of the first queries complete, I want to start serializing.</p>

<p>If I do something like:</p>

<pre><code>Future.sequence(queries)
</code></pre>

<p>It will collect the results of all the queries, but my understanding is that operations like <code>map</code> won't be invoked until all queries complete and all the <code>Thing</code>s must fit into memory at once.</p>

<p>What's the best way to implement a batched stream pipeline using Scala collections and concurrent libraries?</p>