Celery connection drop with AWS ELB and RabbitMQ
<p>In our environment, we use RabbitMQ and Celery on AWS to run tasks in parallel over many nodes.</p>

<p>Recently we turned RabbitMQ into a cluster of 3 nodes, configured a ha policy and added an AWS elastic load balancer (ELB) for port 5672 to all 3 nodes. Our Celery workers and client code all use the ELB DNS as the broker URL. </p>

<p>We have noticed since that change, that waiting for async tasks to finish will throw an exception <code>IOError: Socket closed</code>.</p>

<p>The ELB will shutdown all idle connections after 60 seconds. We have tasks that take few hours to complete. </p>

<p>Setting BROKER_HEARTBEAT to a value lower than 60 solved connection drops on the workers end. But we can't seem to find any setting that will keep the client connection alive. </p>

<p>Is this the correct approach to wait for long running tasks with Celery?</p>

<p>One workaround we haven't tested yet, is to recall the <code>AsyncResult.wait()</code> method until it ends successfully. So for example:</p>

<pre><code>async_result = task.delay(params)

while True:
    try:
        async_result.wait()
        break
    except IOError:
        pass
</code></pre>

<p>We use: </p>

<ol>
<li>RabbitMQ 3.6.5</li>
<li>Celery 3.1.20</li>
<li>Celery backend is pyamqp</li>
<li>Celery results backend is rpc</li>
</ol>