Docker swarm mode load balancing not working as described
<h2>Update</h2>

<p>I believe the culprit is the master who does not appear to be listening on port 7946. <code>netstat</code> shows that 7946 is listening on the nodes, but not the master. When I check the syslogs for the nodes I see the following error</p>

<pre><code>level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
</code></pre>

<h2>Original Post</h2>

<p>I am running a three node Swarm Mode cluster in AWS; one master and two workers. This is <strong>swarm mode</strong> not to be confused with <strong>docker swarm</strong> from pre 1.12. </p>

<p>I created all of the services with docker-machine. Each machine is running Ubuntu 15.10 with Docker 1.12.3.</p>

<pre><code>Linux swarm-master-01 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>

<p>Using the master node I have created a service with the following</p>

<pre><code>docker service create --replicas 1 --name myapp -p 3000 myapp
</code></pre>

<p>When I run <code>docker service ps myapp</code> I get the following output</p>

<pre><code>ID                         NAME     IMAGE         NODE             DESIRED STATE  CURRENT STATE            ERROR
02awst8p9pezgpkfzqgz8z79t  myapp.1  myapp:latest  swarm-node-01    Running        Running 19 minutes ago
</code></pre>

<p>The running task is deployed to swarm-node-01.</p>

<p>I checked the auto-selected port which was published publicly</p>

<pre><code>$ docker service inspect myapp | jq .[].Endpoint.Ports[].PublishedPort
30000
</code></pre>

<p>According to the <a href="https://docs.docker.com/engine/swarm/key-concepts/#load-balancing" rel="nofollow noreferrer">documentation</a>:</p>

<blockquote>
  <p>External components, such as cloud load balancers, can access the service on the PublishedPort of <strong>any node in the cluster whether or not the node is currently running the task for the service</strong>. All nodes in the swarm route ingress connections to a running task instance.</p>
</blockquote>

<p>But when I try to curl the nodes who do not have the task running I'm getting <code>connection refused</code>. </p>

<pre><code>$ curl $(docker-machine ip swarm-node-01):30000/stats
{"uptime":"2016-11-09T14:48:35Z","requestCount":7,"statuses":{"200":7},"pid":1,"open_db_conns":0}

$ curl $(docker-machine ip swarm-node-02):30000/stats
curl: (7) Failed to connect to [the IP] port 30000: Connection refused
</code></pre>

<p><strong><em>note: I scrubbed the IP of node-02</em></strong></p>

<hr>

<p>My Troubleshooting:</p>

<ul>
<li>The nodes are both properly connected to the swarm</li>
<li>Scaling the service up to 5 (which inherently deploys the task to every node) makes curl work on every node, because the task is deployed to every node. </li>
</ul>

<p>UPDATE 1</p>

<p>I initialized the swarm with </p>

<pre><code>docker swarm init --advertise-addr 10.0.0.12:2377 --listen-addr 10.0.0.12:2377
</code></pre>

<p>I checked the syslogs from the nodes and I'm seeing the following errors</p>

<pre><code>level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
</code></pre>

<p>I checked to see if the ingress port was listening and it doesn't seem to be</p>

<pre><code>ubuntu@swarm-master-01:~$ sudo lsof -i :7946
ubuntu@swarm-master-01:~$ cat &lt; /dev/tcp/10.0.0.12/7946
-bash: connect: Connection refused
-bash: /dev/tcp/10.0.0.12/7946: Connection refused
ubuntu@swarm-master-01:~$ cat &lt; /dev/tcp/0.0.0.0/7946
-bash: connect: Connection refused
-bash: /dev/tcp/0.0.0.0/7946: Connection refused
</code></pre>