Java 8 stream and incoming data (Kafka)
<p>I have a queue (it happens to be Kafka but I am not sure that matters) from which I am reading messages. I want to create a stream to represent this data. </p>

<p>My pseudocode to consume a (Kafka) queue looks like this:</p>

<pre><code>List&lt;Message&gt; messages = new ArrayList&lt;&gt;();

while (true) {
    ConsumerRecords&lt;String, Message&gt; records = kafkaConsumer.poll(100);

    messages.add(recordsToMessages(records));

    if (x) {
        break;
    }
}

return messages.stream();
</code></pre>

<p>Using this pseudocode the stream is not returned <strong>until</strong> the while loop is broken, i.e. until all of the queue has been read. </p>

<p>I want to be able to return the stream straight away, i.e. new messages can be added to the stream <strong>after</strong> it has returned.</p>

<p>I feel like I need to use Stream.generate but I am not sure how, or maybe I need a spliterator?</p>

<p>I will also want to close the stream at a later point in the code.</p>

<p>Thanks!</p>