RabbitMQ - Channel shutdown: connection error (SpringXD closes rabbitmq connections repeatedly.)
<p>I had a terrible night trying to figure out what is going on with RabbitMQ and SpringXD, unfortunately without a success.</p>

<p><strong>The problem:
SpringXD closes RabbitMQ connections repeatedly, 
or reports warnings related to the channel cache size.</strong></p>

<p><strong>Fragment from the SpringXD log (during stream initialization/autowiring):</strong></p>

<pre><code> 2016-05-03T07:42:43+0200 1.3.0.RELEASE WARN
 DeploymentsPathChildrenCache-0 listener.SimpleMessageListenerContainer
 - CachingConnectionFactory's channelCacheSize can not be less than the 
 number of concurrentConsumers so it was reset to match: 4

...

 2016-05-03T07:54:17+0200 1.3.0.RELEASE ERROR AMQP Connection
 192.168.120.125:5672 connection.CachingConnectionFactory - Channel shutdown: connection error

 2016-05-03T17:38:58+0200 1.3.0.RELEASE ERROR AMQP Connection
 192.168.120.125:5672 connection.CachingConnectionFactory - Channel shutdown: connection error; protocol method:
 method&lt;connection.close&gt;(reply-code=504, reply-text=CHANNEL_ERROR - 
 second 'channel.open' seen, class-id=20, method-id=10)
</code></pre>

<p><strong>Fragment from the RabbitMQ log:</strong></p>

<pre><code> =WARNING REPORT==== 3-May-2016::08:08:09 === closing AMQP connection &lt;0.22276.61&gt; (192.168.120.125:59350 -&gt; 192.168.120.125:5672): client
 unexpectedly closed TCP connection

 =ERROR REPORT==== 3-May-2016::08:08:11 === closing AMQP connection 0.15409.61&gt; (192.168.120.125:58527 -&gt; 192.168.120.125:5672):
 {writer,send_failed,{error,closed}}
</code></pre>

<blockquote>
  <p>state blocked error is rare </p>
</blockquote>

<pre><code> =ERROR REPORT==== 3-May-2016::17:38:58 === Error on AMQP connection &lt;0.20542.25&gt; (192.168.120.125:59421 -&gt; 192.168.120.125:5672, vhost:
'/', user: 'xd', state: blocked), channel 7: operation channel.open
caused a connection exception channel_error: "second 'channel.open'
 seen"
</code></pre>

<p><strong>My setup (6 nodes)</strong></p>

<pre><code>- springxd 1.3.0 distributed (zookeeper)  
- RabbitMQ 3.6.0, Erlang R16B03-1 cluster


    ackMode:                   AUTO ## or NONE
    autoBindDLQ:               false
    backOffInitialInterval:    1000
    backOffMaxInterval:        10000
    backOffMultiplier:         2.0
    batchBufferLimit:          10000
    batchingEnabled:           false
    batchSize:                 200
    batchTimeout:              5000
    compress:                  false
    concurrency:               4
    deliveryMode:              NON_PERSISTENT ## or PERSISTENT
    durableSubscription:       false
    maxAttempts:               10
    maxConcurrency:            10
    prefix:                    xdbus.
    prefetch:                  1000
    replyHeaderPatterns:       STANDARD_REPLY_HEADERS,*
    republishToDLQ:            false
    requestHeaderPatterns:     STANDARD_REQUEST_HEADERS,*
    requeue:                   true
    transacted:                false
    txSize:                    1000
</code></pre>

<p><strong>spring:
  rabbitmq:</strong></p>

<pre><code>addresses:
priv1:5672,priv2:5672,priv3:5672,
priv4:5672,priv5:5672,priv6:5672

adminAddresses:  
http://priv1:15672, http://priv2:15672, http://priv3:15672, http://priv4:15672, http://priv5:15672,http://priv6:15672

nodes: 
rabbit@priv1,rabbit@priv2,rabbit@priv3,
rabbit@priv4,rabbit@priv5,rabbit@priv6

username: xd
password: xxxx
virtual_host: /
useSSL: false
</code></pre>

<p><strong>ha-xdbus policy:</strong></p>

<pre><code> - ^xdbus\. all  
 - ha-mode: exactly
 - ha-params:   2 
 - queue-master-locator:    min-masters
</code></pre>

<p><strong>Rabbit conf</strong></p>

<pre><code>[
 {rabbit, 
[
     {tcp_listeners, [5672]},
     {queue_master_locator, "min-masters"}
]
}
].
</code></pre>

<p><strong>When ackMode is NONE the following happens:</strong></p>

<p>Eventually the number of consumers drop to zero and I have a zombie streams that don't recover from that state, which in turn causes unwanted queueing.</p>

<p><strong>When ackMode is AUTO the following happens:</strong></p>

<p>Some messages left un-acked forever.</p>

<p><strong>SpringXD streams and durable queues</strong></p>

<p>Rabbit module is being used as source or sink, no custom autowiring.</p>

<p>Typical stream definitions are as follows:</p>

<p>Ingestion:</p>

<pre><code>event_generator | rabbit --mappedRequestHeaders=XDRoutingKey --routingKey='headers[''XDRoutingKey'']'
</code></pre>

<p>Processing/Sink:</p>

<pre><code>rabbit --queues='xdbus.INQUEUE-A' | ENRICHMENT-PROCESSOR-A | elastic-sink
rabbit --queues='xdbus.INQUEUE-B' | ENRICHMENT-PROCESSOR-B | elastic-sink
</code></pre>

<p>xdbus.INQUEUE-xxx are manually created from the Rabbit admin GUI. (durable)</p>

<p><strong>GLOBAL statistics (from the RabbitMQ Admin)</strong></p>

<ul>
<li>Connections: 190 </li>
<li>Channels: 2263 (Channel cache problem perhaps ?)</li>
<li>Exchanges: 20 </li>
<li>Queues: 120 </li>
<li>Consumers : 1850</li>
</ul>

<p><strong>Finally:</strong></p>

<p>I would appreciate if someone could answer what is wrong with the configuration (I am pretty sure the network is performing well, so there are no network problems  and there is no problem related to  max open files limitation).</p>

<p>Message rates vary from 2K/sec to max 30k/sec which is relative small load.</p>

<p>Thanks!</p>

<p>Ivan</p>