Partition and Consumer location after rebalnce of Apache Kafka
<p>I want to achieve the following effect.
Let's say I have 9 partitions spread out over 3 workers. that would look something like this:</p>

<p><a href="https://i.stack.imgur.com/kXrTc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/kXrTc.png" alt="Before repartition"></a></p>

<p>What I want to do, is that when a new machine joins the consumer group.
thus adding a new consumer. I want the rebalancing to only move as few partitions as possible to the new consumer.</p>

<p>Like so: 
See how partition 3 6 9 ended up on the new consumer.
While the other consumers retained most of their existing partitions.</p>

<p><a href="https://i.stack.imgur.com/PHTLp.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PHTLp.png" alt="After repartition"></a></p>

<p>Is this possible?
What I want to do is to build stateful systems behind Kafka.
Which I can already do, but when a consumer joins or leaves, the default behavior is just to disconnect everything and then do a rebalance from scratch.
Thus making a lot of partitions to change consumers.
Does any of this make sense?</p>