{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1245","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1245/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1245/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1245/events","html_url":"https://github.com/hashicorp/consul/issues/1245","id":107089384,"number":1245,"title":"Joining multiple datacenters, failed but ports open","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-17T22:33:47Z","updated_at":"2015-09-21T17:16:45Z","closed_at":null,"body":"Hello,\r\n\r\nI have set up multiple datacenters in AWS in regions us-west-2, us-east-1, and eu-west-1, each running 5 consul servers.\r\n\r\nI ssh'd into one instance in eu-west-1 and ran `consul join -wan 10.179.3.152` which is the private IP of a consul server in us-west-2. I ssh'd into another instance in us-east-1 and ran `consul join -wan 10.179.3.152`. So now all three regions should be connected.\r\n\r\nWhen I navigate to 10.179.3.152:8500 in a browser, it brings me up to a new screen that lets me choose a datacenter, however it just sits in the loading screen for the other datacenters and reports this in the logs:\r\n```\r\n2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n    2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n    2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n```\r\nIt loads fine for us-west-2, which is where 10.179.3.8500 exists in.\r\n\r\nI ran `consul members -wan` and see this:\r\n\r\n```\r\nNode                        Address             Status  Type    Build  Protocol  DC\r\nip-10-179-3-152.us-west-2   10.179.3.152:8302   alive   server  0.5.2  2         us-west-2\r\nip-10-171-154-21.us-east-1  10.171.154.21:8302  alive   server  0.5.2  2         us-east-1\r\nip-10-172-4-9.eu-west-1     10.172.4.9:8302     alive   server  0.5.2  2         eu-west-1\r\n```\r\n\r\nLooks good.\r\n\r\nThen I checked the logs for 10.179.3.152 and see this:\r\n```\r\n.\r\n.\r\n.\r\n    2015/09/17 22:20:32 [ERR] memberlist: Push/Pull with ip-10-171-154-21.us-east-1 failed: read tcp 10.171.154.21:8302: i/o timeout\r\n    2015/09/17 22:20:33 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:20:38 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:20:43 [INFO] memberlist: Marking ip-10-171-154-21.us-east-1 as failed, suspect timeout reached\r\n    2015/09/17 22:20:43 [INFO] serf: EventMemberFailed: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:20:43 [INFO] consul: removing server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:20:46 [WARN] memberlist: Refuting a suspect message (from: ip-10-172-4-9.eu-west-1)\r\n    2015/09/17 22:20:48 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:20:58 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:21:00 [INFO] serf: EventMemberJoin: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:21:00 [INFO] consul: adding server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:21:08 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:21:13 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:21:18 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:21:28 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:21:33 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:21:38 [INFO] memberlist: Marking ip-10-172-4-9.eu-west-1 as failed, suspect timeout reached\r\n    2015/09/17 22:21:38 [INFO] serf: EventMemberFailed: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:21:38 [INFO] consul: removing server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:21:38 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:21:43 [INFO] memberlist: Marking ip-10-171-154-21.us-east-1 as failed, suspect timeout reached\r\n    2015/09/17 22:21:43 [INFO] serf: EventMemberFailed: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:21:43 [INFO] consul: removing server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:21:43 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:21:53 [INFO] serf: EventMemberJoin: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:21:53 [INFO] consul: adding server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:22:00 [INFO] serf: EventMemberJoin: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:22:00 [INFO] consul: adding server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:22:03 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:08 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:13 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:22:18 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:23 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:28 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:22:33 [INFO] memberlist: Marking ip-10-172-4-9.eu-west-1 as failed, suspect timeout reached\r\n    2015/09/17 22:22:33 [INFO] serf: EventMemberFailed: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:22:33 [INFO] consul: removing server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:22:33 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:34 [INFO] serf: attempting reconnect to ip-10-172-4-9.eu-west-1 10.172.4.9:8302\r\n    2015/09/17 22:22:35 [INFO] serf: EventMemberJoin: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:22:35 [INFO] consul: adding server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:22:38 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:22:43 [INFO] memberlist: Marking ip-10-171-154-21.us-east-1 as failed, suspect timeout reached\r\n    2015/09/17 22:22:43 [INFO] serf: EventMemberFailed: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:22:43 [INFO] consul: removing server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:22:43 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:53 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:22:58 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:23:08 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:23:12 [WARN] memberlist: Refuting a dead message (from: ip-10-172-4-9.eu-west-1)\r\n    2015/09/17 22:23:13 [INFO] memberlist: Marking ip-10-172-4-9.eu-west-1 as failed, suspect timeout reached\r\n    2015/09/17 22:23:13 [INFO] serf: EventMemberFailed: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:23:13 [INFO] consul: removing server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:23:18 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:23:25 [INFO] agent.rpc: Accepted client: 127.0.0.1:41986\r\n    2015/09/17 22:23:31 [INFO] agent.rpc: Accepted client: 127.0.0.1:41987\r\n    2015/09/17 22:23:31 [INFO] agent.rpc: Accepted client: 127.0.0.1:41988\r\n    2015/09/17 22:23:32 [INFO] agent.rpc: Accepted client: 127.0.0.1:41989\r\n    2015/09/17 22:23:32 [INFO] serf: EventMemberJoin: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:23:32 [INFO] consul: adding server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:23:33 [INFO] agent.rpc: Accepted client: 127.0.0.1:41990\r\n    2015/09/17 22:23:33 [INFO] agent.rpc: Accepted client: 127.0.0.1:41991\r\n    2015/09/17 22:23:34 [INFO] agent.rpc: Accepted client: 127.0.0.1:41992\r\n    2015/09/17 22:23:35 [INFO] agent.rpc: Accepted client: 127.0.0.1:41993\r\n    2015/09/17 22:23:36 [INFO] agent.rpc: Accepted client: 127.0.0.1:41994\r\n    2015/09/17 22:23:38 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:23:47 [INFO] agent.rpc: Accepted client: 127.0.0.1:41996\r\n    2015/09/17 22:23:48 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:23:53 [INFO] serf: EventMemberJoin: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:23:53 [INFO] consul: adding server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:23:53 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:23:58 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:24:03 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:24:08 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:24:13 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:24:18 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:24:23 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:24:26 [ERR] yamux: Failed to read header: read tcp 10.172.4.9:8300: connection timed out\r\n    2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n    2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n    2015/09/17 22:24:26 [ERR] http: Request /v1/internal/ui/nodes?dc=eu-west-1&token=<hidden>, error: rpc error: unexpected EOF\r\n    2015/09/17 22:24:28 [INFO] memberlist: Marking ip-10-171-154-21.us-east-1 as failed, suspect timeout reached\r\n    2015/09/17 22:24:28 [INFO] serf: EventMemberFailed: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:24:28 [INFO] consul: removing server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:24:28 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:24:30 [INFO] serf: EventMemberJoin: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:24:30 [INFO] consul: adding server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:24:38 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:24:43 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:24:48 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:24:58 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:25:02 [ERR] memberlist: Push/Pull with ip-10-171-154-21.us-east-1 failed: read tcp 10.171.154.21:8302: i/o timeout\r\n    2015/09/17 22:25:03 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:25:08 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:25:13 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:25:18 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:25:18 [INFO] memberlist: Marking ip-10-171-154-21.us-east-1 as failed, suspect timeout reached\r\n    2015/09/17 22:25:18 [INFO] serf: EventMemberFailed: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:25:18 [INFO] consul: removing server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:25:19 [INFO] memberlist: Marking ip-10-172-4-9.eu-west-1 as failed, suspect timeout reached\r\n    2015/09/17 22:25:19 [INFO] serf: EventMemberFailed: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:25:19 [INFO] consul: removing server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:25:23 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:25:30 [INFO] serf: EventMemberJoin: ip-10-171-154-21.us-east-1 10.171.154.21\r\n    2015/09/17 22:25:30 [INFO] consul: adding server ip-10-171-154-21.us-east-1 (Addr: 10.171.154.21:8300) (DC: us-east-1)\r\n    2015/09/17 22:25:34 [INFO] serf: attempting reconnect to ip-10-172-4-9.eu-west-1 10.172.4.9:8302\r\n    2015/09/17 22:25:35 [INFO] serf: EventMemberJoin: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:25:35 [INFO] consul: adding server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:25:43 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:25:48 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:25:54 [WARN] memberlist: Refuting a suspect message (from: ip-10-172-4-9.eu-west-1)\r\n    2015/09/17 22:25:58 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:26:03 [INFO] agent.rpc: Accepted client: 127.0.0.1:42004\r\n    2015/09/17 22:26:03 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:26:08 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n    2015/09/17 22:26:13 [INFO] memberlist: Marking ip-10-172-4-9.eu-west-1 as failed, suspect timeout reached\r\n    2015/09/17 22:26:13 [INFO] serf: EventMemberFailed: ip-10-172-4-9.eu-west-1 10.172.4.9\r\n    2015/09/17 22:26:13 [INFO] consul: removing server ip-10-172-4-9.eu-west-1 (Addr: 10.172.4.9:8300) (DC: eu-west-1)\r\n    2015/09/17 22:26:13 [INFO] memberlist: Suspect ip-10-172-4-9.eu-west-1 has failed, no acks received\r\n    2015/09/17 22:26:16 [WARN] memberlist: Refuting a suspect message (from: ip-10-171-154-21.us-east-1)\r\n    2015/09/17 22:26:18 [INFO] memberlist: Suspect ip-10-171-154-21.us-east-1 has failed, no acks received\r\n.\r\n.\r\n.\r\n```\r\n\r\nSo I'm thinking ok, I must have one of the ports block. So I ran netcat to check the SERF ports and those succeed:\r\n```\r\nubuntu@ip-10-179-3-152:~$ netcat -zvu 10.172.4.9 8301\r\nConnection to 10.172.4.9 8301 port [udp/*] succeeded!\r\nubuntu@ip-10-179-3-152:~$ netcat -zvu 10.172.4.9 8302\r\nConnection to 10.172.4.9 8302 port [udp/*] succeeded!\r\nubuntu@ip-10-179-3-152:~$ netcat -zv 10.172.4.9 8302\r\nConnection to 10.172.4.9 8302 port [tcp/*] succeeded!\r\nubuntu@ip-10-179-3-152:~$ netcat -zv 10.172.4.9 8301\r\nConnection to 10.172.4.9 8301 port [tcp/*] succeeded!\r\n```\r\nRan the same against the us-east-1 box, and those succeed as well.\r\n\r\nNow I'm not sure what to do because the ports are open, and the communication appears open, but Consul doesn't want to communicate. Am I missing a port?\r\n\r\n\r\n\r\nAlso, if I navigate to the UI on any other consul server in the cluster, it doesn't see the new datacenters? I thought the WAN joins were supposed to replicate to the other servers as well?"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/142046939","html_url":"https://github.com/hashicorp/consul/issues/1245#issuecomment-142046939","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1245","id":142046939,"user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"created_at":"2015-09-21T17:16:45Z","updated_at":"2015-09-21T17:16:45Z","body":"They're always in a failed state. It looks like its going to work in the beginning because you can see the multi-datacenter screen in the beginning asking you to select a datacenter. If you select the datacenter that the server you're accessing through exists in, it loads fine. If you click on any other datacenter, the screen changes to the empty rectangle box with an image icon at the top left, and \"Loading...\" right below.\r\n\r\nAfter about a minute it changes to a 504 timeout.\r\n\r\n```\r\nHTTP error code from Consul: 504 GATEWAY_TIMEOUT\r\n\r\nThis is an error page for the Consul web UI. You may have visited a URL that is loading an unknown resource, so you can try going back to the root.\r\n\r\nOtherwise, please report any unexpected issues on the GitHub page.\r\n```"}}