Spark Streaming + Kinesis : Initial record consumption
<p>I'm feeding Spark streaming with a Kinesis Stream.
My project is using 1s batches,
during the first batches (the queue contains a few million items, and task is told to start from the beginning of the stream)
spark streaming start consuming batches of 10K records.
This happens every 10/20s.</p>

<p>i.e:</p>

<p><code>
t0 -&gt; records : 0
t1 -&gt; records : 0
.....
t10 -&gt; records: 10.000 -&gt; total process time is 0.8s (lower than batch time)
t11 -&gt; recods : 0
..
t15 -&gt;records : 0
..
t20 -&gt; records: 10.000
</code></p>

<p>this beaviour occurs until spark cathces up with the top of the stream. After htat every batch will process elements every second.</p>

<p>It feels like at the start point it should consistently process a number of records per batch, without having htat high numberof batches processing no records.</p>

<p>Any setting that I'm ignoring? is the described behaviour expected?</p>

<p><a href="https://i.stack.imgur.com/0QJDv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0QJDv.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/mwevr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mwevr.png" alt="enter image description here"></a></p>