Stop celery workers to consume from all queues
<p>Cheers,</p>

<p>I have a celery setup running in a production environment (on Linux) where I need to consume two different task types from two dedicated queues (one for each). The problem that arises is, that all workers are always bound to <strong>both queues</strong>, even when I specify them to only consume from one of them.</p>

<h2>TL;DR</h2>

<ul>
<li>Celery running with 2 queues</li>
<li>Messages are published in correct queue as designed</li>
<li>Workers keep consuming both queues</li>
<li>Leads to deadlock</li>
</ul>

<h2>General Information</h2>

<p>Think of my two different task types as a hierarchical setup:</p>

<ul>
<li><p>A <strong>task</strong> is a regular celery task that may take quite some time, because it dynamically dispatches other celery tasks and may be required to chain through their respective results</p></li>
<li><p>A <strong>node</strong> is a dynamically dispatched sub-task, which also is a regular celery task but itself can be considered an atomic unit.</p></li>
</ul>

<p>My <strong>task</strong> thus can be a more complex setup of <strong>node</strong>s where the results of one or more nodes serves as input for one or more subsequent nodes, and so on. Since my <strong>task</strong>s can take longer and will only finish when all their <strong>node</strong>s have been deployed, it is <em>essential</em> that they are handled by dedicated workers to keep a sufficient number of workers free to consume the <strong>node</strong>s. Otherwise, this could lead to the system being stuck, when a lot of <strong>task</strong>s are dispatched, each consumed by another worker, and their respective <strong>node</strong>s are only queued but will never be consumed, because all workers are blocked.</p>

<p>If this is a bad design in general, please make any propositions on how I can improve it. I did not yet manage to build one of these processes using celery's built-in canvas primitives. Help me, if you can?!</p>

<h2>Configuration/Setup</h2>

<p>I run celery with <code>amqp</code> and have set up the following queues and routes in the celery configuration:</p>

<pre><code>CELERY_QUERUES = (
    Queue('prod_nodes', Exchange('prod'), routing_key='prod.node'),
    Queue('prod_tasks', Exchange('prod'), routing_key='prod.task')
)

CELERY_ROUTES = (
    'deploy_node': {'queue': 'prod_nodes', 'routing_key': 'prod.node'},
    'deploy_task': {'queue': 'prod_tasks', 'routing_key': 'prod.task'}
)
</code></pre>

<p>When I launch my workers, I issue a call similar to the following:</p>

<pre><code>celery multi start w_task_01 w_node_01 w_node_02 -A my.deployment.system \
  -E -l INFO -P gevent -Q:1 prod_tasks -Q:2-3 prod_nodes -c 4 --autoreload \
  --logfile=/my/path/to/log/%N.log --pidfile=/my/path/to/pid/%N.pid
</code></pre>

<h2>The Problem</h2>

<p>My queue and routing setup seems to work properly, as I can see messages being correctly queued in the <em>RabbitMQ Management</em> web UI.</p>

<p>However, all workers always consume celery tasks from <strong>both</strong> queues. I can see this when I start and open up the <em>flower</em> web UI and inspect one of the deployed tasks, where e.g. <em>w_node_01</em> starts consuming messages from the <em>prod_tasks</em> queue, even though it shouldn't.</p>

<p>The <em>RabbitMQ Management</em> web UI furthermore tells me, that all started workers are set up as <em>consumers</em> for both queues.</p>

<h2>Thus, I ask you...</h2>

<p>... what did I do wrong?</p>

<p>Where is the issue with my setup or worker start call; How can I circumvent the problem of workers always consuming from both queues; Do I really have to make additional settings during runtime (what I certainly do not want)?</p>

<p>Thanks for your time and answers!</p>