How can I scale Kafka consumers?
<p>I'm reading the Kafka documentation and noticed the following line: </p>

<blockquote>
  <p>Note however that there cannot be more consumer instances in a consumer group than partitions.</p>
</blockquote>

<p>Hmm.  How can I auto-scale this?</p>

<p>For example let's say I have a messaging system with hi/lo priorities, so I create a topic for messages and partitions for hi and lo priority messages.</p>

<p>If this was RabbitMQ, I'd have an auto-scalable group of consumers assigned to each partition, like this:</p>

<p><a href="https://i.stack.imgur.com/74VYo.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/74VYo.jpg" alt="enter image description here"></a></p>

<p>If I understand the Kafka model I can't have >1 consumer per partition in a consumer group, so that picture doesn't work for Kafka, right?</p>

<p>Ok, so what about >1 consumer groups like this:</p>

<p><a href="https://i.stack.imgur.com/1cTcf.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/1cTcf.jpg" alt="enter image description here"></a></p>

<p>That get's around Kafka's limitation but... If I understand how this works both consumer groups would be pulling from a partition, for example msg.hi, with their own offsets so neither would know about the other--meaning messages would likely be delivered twice!</p>

<p>How can I achieve the capability I had in the Rabbit design w/Kafka and still maintain the "queue-ness" of the behavior (I don't want to send a message twice)?  What am I missing?</p>