RabbitMQ consumer utilization zero under high load
<p>We use spring amqp based listener container - with about 40 consumers, prefetch count 1. The message ttl is about 60sec before it gets to dead letter queue.</p>

<p>The operation performed by each consumer is a database update - which is slower than the rate at which the messages arrive into the queue.</p>

<p>After sometime, the number of messages pile up in queue the consumer utilization is down to zero. I was under the impression that the consumers are blocked on database. However if I look at the thread dump all the consumers are in wait state on rabbit mq - no messages are being processed.</p>

<pre><code>    "SimpleAsyncTaskExecutor-7" #51 prio=5 os_prio=0 tid=0x00007fcb01ad0800 nid=0x58f7 waiting on condition [0x00007fcae5af1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  &lt;0x00000000854c30c8&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:390)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1097)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1086)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1100(SimpleMessageListenerContainer.java:93)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1203)
    at java.lang.Thread.run(Thread.java:745)
</code></pre>

<p>The queue also gets into flow state. </p>

<p>Not sure why the queue message processing is stopped. I understand publish to it being restricted.</p>

<p>Any suggestions would help.</p>