{"action":"created","issue":{"url":"https://api.github.com/repos/zettio/weave/issues/416","labels_url":"https://api.github.com/repos/zettio/weave/issues/416/labels{/name}","comments_url":"https://api.github.com/repos/zettio/weave/issues/416/comments","events_url":"https://api.github.com/repos/zettio/weave/issues/416/events","html_url":"https://github.com/zettio/weave/issues/416","id":58796433,"number":416,"title":"keep losing connectivity between nodes","user":{"login":"bmurphy1976","id":270488,"avatar_url":"https://avatars.githubusercontent.com/u/270488?v=3","gravatar_id":"","url":"https://api.github.com/users/bmurphy1976","html_url":"https://github.com/bmurphy1976","followers_url":"https://api.github.com/users/bmurphy1976/followers","following_url":"https://api.github.com/users/bmurphy1976/following{/other_user}","gists_url":"https://api.github.com/users/bmurphy1976/gists{/gist_id}","starred_url":"https://api.github.com/users/bmurphy1976/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmurphy1976/subscriptions","organizations_url":"https://api.github.com/users/bmurphy1976/orgs","repos_url":"https://api.github.com/users/bmurphy1976/repos","events_url":"https://api.github.com/users/bmurphy1976/events{/privacy}","received_events_url":"https://api.github.com/users/bmurphy1976/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-24T20:08:38Z","updated_at":"2015-02-25T17:03:51Z","closed_at":null,"body":"I'm really struggling with this.  We are constantly losing connectivity between nodes in the weave cluster.  Here's an example.  On one host, we have a container running a simple nodejs service:\r\n\r\n```\r\nroot@a6e66c7b4052:/# ifconfig eth0\r\neth0      Link encap:Ethernet  HWaddr 02:42:c0:a8:02:11\r\n          inet addr:192.168.2.17  Bcast:0.0.0.0  Mask:255.255.0.0\r\n          inet6 addr: fe80::42:c0ff:fea8:211/64 Scope:Link\r\n          UP BROADCAST RUNNING  MTU:9001  Metric:1\r\n          RX packets:6835 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:134 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:0\r\n          RX bytes:394247 (394.2 KB)  TX bytes:13952 (13.9 KB)\r\n\r\nroot@a6e66c7b4052:/# time curl http://127.0.0.1:3000/\r\nCannot GET /\r\n\r\nreal\t0m0.023s\r\nuser\t0m0.007s\r\nsys\t0m0.007s\r\n```\r\n\r\nRequests to port 3000 are very consistent and fast.  On another server, requests to the same service are sometimes fast, sometimes excruciatingly slow, and sometimes never complete: \r\n\r\n```\r\nroot@eeafe17b777b:/# ifconfig eth0\r\neth0      Link encap:Ethernet  HWaddr 02:42:c0:a8:10:14\r\n          inet addr:192.168.16.20  Bcast:0.0.0.0  Mask:255.255.0.0\r\n          inet6 addr: fe80::42:c0ff:fea8:1014/64 Scope:Link\r\n          UP BROADCAST RUNNING  MTU:1500  Metric:1\r\n          RX packets:124218 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:86351 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:0\r\n          RX bytes:74246606 (74.2 MB)  TX bytes:69870847 (69.8 MB)\r\n\r\nroot@eeafe17b777b:/# time curl http://192.168.2.17:3000/\r\nCannot GET /\r\n\r\nreal\t0m31.077s\r\nuser\t0m0.008s\r\nsys\t0m0.010s\r\nroot@eeafe17b777b:/# time curl http://192.168.2.17:3000/\r\nCannot GET /\r\n\r\nreal\t0m0.027s\r\nuser\t0m0.003s\r\nsys\t0m0.012s\r\n```\r\n\r\nThese hosts do not experience this issue when querying the mapped public port, it only happens over the weave network and this is not limited to just this host and service.  So far it appears to be happening on any service that is accessed over the weave network.\r\n\r\nOccasionally the weaver process uses an excessive amount of CPU but this does not appear to correlate with the responsiveness issues.\r\n\r\n\r\nWeave status for the calling node:\r\n\r\n```\r\n# weave status\r\nweave router 0.9.0\r\nEncryption on\r\nOur name is 7a:8f:17:40:9e:ce\r\nSniffing traffic on &{282 65535 ethwe 6e:d1:c4:d1:36:02 up|broadcast|multicast}\r\nMACs:\r\n02:42:c0:a8:02:01 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:00:58.128549856 +0000 UTC)\r\n02:42:c0:a8:04:47 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.571293935 +0000 UTC)\r\n02:42:c0:a8:04:1b -> 7a:87:ef:4c:d7:5e (2015-02-24 19:58:04.311527217 +0000 UTC)\r\n02:42:c0:a8:04:32 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.826728747 +0000 UTC)\r\n02:42:c0:a8:10:15 -> 7a:8f:17:40:9e:ce (2015-02-24 20:06:14.591745614 +0000 UTC)\r\n02:42:c0:a8:04:30 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:03:05.55304924 +0000 UTC)\r\n02:42:c0:a8:04:1d -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.600979829 +0000 UTC)\r\n02:42:c0:a8:04:35 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:07.276001546 +0000 UTC)\r\n02:42:c0:a8:04:34 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:08.111858588 +0000 UTC)\r\n02:42:c0:a8:0f:07 -> 7a:b5:2d:25:d1:cf (2015-02-24 20:06:09.92084675 +0000 UTC)\r\n02:42:c0:a8:02:11 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:04:18.778002421 +0000 UTC)\r\n02:42:c0:a8:04:18 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.555536732 +0000 UTC)\r\n02:42:c0:a8:04:1f -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.157176389 +0000 UTC)\r\n02:42:c0:a8:04:21 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.417463818 +0000 UTC)\r\n02:42:c0:a8:04:26 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:08.59936662 +0000 UTC)\r\n02:42:c0:a8:04:33 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.639964132 +0000 UTC)\r\n7a:8c:a1:e4:43:13 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:05:30.023810055 +0000 UTC)\r\n02:42:c0:a8:04:1e -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.095454332 +0000 UTC)\r\n02:42:c0:a8:04:28 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.268388573 +0000 UTC)\r\n02:42:c0:a8:04:37 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:20.315866829 +0000 UTC)\r\n02:42:c0:a8:04:45 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.059219027 +0000 UTC)\r\n02:42:c0:a8:04:2e -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:04.970775667 +0000 UTC)\r\n02:42:c0:a8:0e:05 -> 7a:8b:d8:d8:4a:88 (2015-02-24 20:06:02.832582154 +0000 UTC)\r\n02:42:c0:a8:04:23 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.890104282 +0000 UTC)\r\n7a:21:54:68:1a:41 -> 7a:21:54:68:1a:41 (2015-02-24 20:05:06.71560369 +0000 UTC)\r\n7a:b5:2d:25:d1:cf -> 7a:b5:2d:25:d1:cf (2015-02-24 20:02:39.180907712 +0000 UTC)\r\n02:42:c0:a8:04:29 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.28517701 +0000 UTC)\r\n02:42:c0:a8:04:2f -> 7a:87:ef:4c:d7:5e (2015-02-24 20:03:04.402336856 +0000 UTC)\r\n02:42:c0:a8:04:44 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:07.4648512 +0000 UTC)\r\n02:42:c0:a8:04:2b -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.343252747 +0000 UTC)\r\n02:42:c0:a8:04:16 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:56:03.727459999 +0000 UTC)\r\n02:42:c0:a8:04:19 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.555787957 +0000 UTC)\r\n02:42:c0:a8:04:1c -> 7a:87:ef:4c:d7:5e (2015-02-24 19:58:04.433702419 +0000 UTC)\r\n02:42:c0:a8:04:22 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.952874273 +0000 UTC)\r\n02:42:c0:a8:04:46 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.416357461 +0000 UTC)\r\n7a:87:ef:4c:d7:5e -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:14.330666322 +0000 UTC)\r\n02:42:c0:a8:04:43 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:07.239534583 +0000 UTC)\r\n02:42:c0:a8:04:15 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:56:03.631301471 +0000 UTC)\r\n02:42:c0:a8:04:2d -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:05.176842007 +0000 UTC)\r\n02:42:c0:a8:0d:07 -> 7a:02:0f:ac:19:f4 (2015-02-24 20:06:04.826287896 +0000 UTC)\r\n02:42:c0:a8:04:31 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.283221763 +0000 UTC)\r\n02:42:c0:a8:04:17 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.301055143 +0000 UTC)\r\n02:42:c0:a8:04:42 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:07.71504102 +0000 UTC)\r\n02:42:c0:a8:10:14 -> 7a:8f:17:40:9e:ce (2015-02-24 20:06:05.736288262 +0000 UTC)\r\n02:42:c0:a8:04:27 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:08.661867098 +0000 UTC)\r\n02:42:c0:a8:04:20 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.482635762 +0000 UTC)\r\n02:42:c0:a8:04:38 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:05.737246212 +0000 UTC)\r\n02:42:c0:a8:04:63 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:58:10.319785295 +0000 UTC)\r\n02:42:c0:a8:04:1a -> 7a:87:ef:4c:d7:5e (2015-02-24 19:58:04.127775525 +0000 UTC)\r\n02:42:c0:a8:04:2a -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.373395947 +0000 UTC)\r\n02:42:c0:a8:04:36 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:07.838330623 +0000 UTC)\r\n02:42:c0:a8:04:25 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.998066837 +0000 UTC)\r\n02:42:c0:a8:04:24 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.946166343 +0000 UTC)\r\n02:42:c0:a8:04:2c -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:04.789619989 +0000 UTC)\r\nPeers:\r\nPeer 7a:8f:17:40:9e:ce (v19) (UID 10167509669896236954)\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783]\r\nPeer 7a:87:ef:4c:d7:5e (v137) (UID 4530646544833339324)\r\n   -> 7a:b5:2d:25:d1:cf [192.168.4.254:39140]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.4.254:34091]\r\n   -> 7a:8f:17:40:9e:ce [192.168.4.254:49022]\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.4.254:48716]\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nPeer 7a:8b:d8:d8:4a:88 (v890) (UID 3299225439164496535)\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nPeer 7a:8c:a1:e4:43:13 (v1660) (UID 10303124246416533281)\r\n   -> 7a:b5:2d:25:d1:cf [192.168.2.254:60685]\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.2.254:59422]\r\n   -> 7a:8f:17:40:9e:ce [192.168.2.254:51743]\r\n   -> 7a:87:ef:4c:d7:5e [192.168.2.254:43466]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.2.254:46898 (unestablished)]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nPeer 7a:02:0f:ac:19:f4 (v897) (UID 6948683680276824273)\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783 (unestablished)]\r\n   -> 7a:87:ef:4c:d7:5e [192.168.4.254:6783]\r\nPeer 7a:21:54:68:1a:41 (v298) (UID 16892061667268854472)\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.1.254:48383]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.1.254:37885]\r\n   -> 7a:87:ef:4c:d7:5e [192.168.1.254:53988]\r\n   -> 7a:b5:2d:25:d1:cf [192.168.1.254:51106]\r\n   -> 7a:8f:17:40:9e:ce [192.168.1.254:45757]\r\n   -> 7a:8c:a1:e4:43:13 [192.168.1.254:50783]\r\nPeer 7a:b5:2d:25:d1:cf (v1432) (UID 2786178580184412291)\r\n   -> 7a:8c:a1:e4:43:13 [192.168.2.254:6783]\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nRoutes:\r\nunicast:\r\n7a:8f:17:40:9e:ce -> 00:00:00:00:00:00\r\n7a:87:ef:4c:d7:5e -> 7a:87:ef:4c:d7:5e\r\n7a:21:54:68:1a:41 -> 7a:21:54:68:1a:41\r\n7a:8c:a1:e4:43:13 -> 7a:8c:a1:e4:43:13\r\n7a:8b:d8:d8:4a:88 -> 7a:21:54:68:1a:41\r\n7a:02:0f:ac:19:f4 -> 7a:21:54:68:1a:41\r\n7a:b5:2d:25:d1:cf -> 7a:21:54:68:1a:41\r\nbroadcast:\r\n7a:8f:17:40:9e:ce -> [7a:87:ef:4c:d7:5e 7a:21:54:68:1a:41 7a:8c:a1:e4:43:13]\r\n7a:87:ef:4c:d7:5e -> []\r\n7a:8b:d8:d8:4a:88 -> []\r\n7a:8c:a1:e4:43:13 -> []\r\n7a:02:0f:ac:19:f4 -> []\r\n7a:21:54:68:1a:41 -> []\r\n7a:b5:2d:25:d1:cf -> []\r\nReconnects:\r\n192.168.2.254:46898 (next try at 2015-02-24 20:06:24.173030984 +0000 UTC)\r\n192.168.1.254:51106 (next try at 2015-02-24 20:09:07.582903676 +0000 UTC)\r\n192.168.2.254:46828 (trying since 2015-02-24 20:04:48.354378388 +0000 UTC)\r\n192.168.2.254:6783 (next try at 2015-02-24 20:12:58.19520751 +0000 UTC)\r\n192.168.2.254:60685 (next try at 2015-02-24 20:06:21.316287575 +0000 UTC)\r\n192.168.4.254:48716 (next try at 2015-02-24 20:08:08.07571165 +0000 UTC)\r\n192.168.1.254:6783 (next try at 2015-02-24 20:07:20.248102243 +0000 UTC)\r\n192.168.2.254:59422 (next try at 2015-02-24 20:06:19.769932448 +0000 UTC)\r\n192.168.1.254:48383 (next try at 2015-02-24 20:06:33.983130298 +0000 UTC)\r\n192.168.4.254:34091 (next try at 2015-02-24 20:06:44.247926547 +0000 UTC)\r\n192.168.1.254:37885 (next try at 2015-02-24 20:13:04.250763677 +0000 UTC)\r\n192.168.4.254:6783 (next try at 2015-02-24 20:11:32.228623543 +0000 UTC)\r\n192.168.4.254:39140 (next try at 2015-02-24 20:14:05.9912991 +0000 UTC)\r\n```\r\n\r\nWeave status for the hosting node:\r\n\r\n```\r\nweave router 0.9.0\r\nEncryption on\r\nOur name is 7a:8c:a1:e4:43:13\r\nSniffing traffic on &{47786 65535 ethwe 2e:79:3e:cb:aa:ce up|broadcast|multicast}\r\nMACs:\r\n02:42:c0:a8:0d:07 -> 7a:02:0f:ac:19:f4 (2015-02-24 20:06:17.294747716 +0000 UTC)\r\n02:42:c0:a8:04:20 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.478176906 +0000 UTC)\r\n02:42:c0:a8:04:23 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.885874861 +0000 UTC)\r\n02:42:c0:a8:04:2f -> 7a:87:ef:4c:d7:5e (2015-02-24 20:03:04.398865883 +0000 UTC)\r\n02:42:c0:a8:04:24 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.913082047 +0000 UTC)\r\n02:42:c0:a8:04:31 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.275366143 +0000 UTC)\r\n02:42:c0:a8:04:2e -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:04.967069414 +0000 UTC)\r\n7a:21:54:68:1a:41 -> 7a:21:54:68:1a:41 (2015-02-24 20:05:47.362959286 +0000 UTC)\r\n02:42:c0:a8:04:18 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.553458508 +0000 UTC)\r\n02:42:c0:a8:04:1e -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.090809986 +0000 UTC)\r\n02:42:c0:a8:04:1f -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.152775233 +0000 UTC)\r\n02:42:c0:a8:04:37 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:20.315127984 +0000 UTC)\r\n02:42:c0:a8:04:1d -> 7a:87:ef:4c:d7:5e (2015-02-24 19:59:04.596991628 +0000 UTC)\r\n02:42:c0:a8:04:33 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.636061151 +0000 UTC)\r\n02:42:c0:a8:04:34 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:12.038527646 +0000 UTC)\r\n02:42:c0:a8:04:32 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:04:04.822944726 +0000 UTC)\r\n02:42:c0:a8:04:22 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.948263032 +0000 UTC)\r\n02:42:c0:a8:04:46 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.411997493 +0000 UTC)\r\n02:42:c0:a8:04:21 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.413515576 +0000 UTC)\r\n02:42:c0:a8:04:27 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:08.657890969 +0000 UTC)\r\n02:42:c0:a8:04:29 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.280407831 +0000 UTC)\r\n02:42:c0:a8:04:2d -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:05.172849962 +0000 UTC)\r\n02:42:c0:a8:04:19 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.553656584 +0000 UTC)\r\n02:42:c0:a8:04:36 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:12.790461878 +0000 UTC)\r\n7a:8c:a1:e4:43:13 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:05:46.909717473 +0000 UTC)\r\n02:42:c0:a8:04:63 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:18.067885456 +0000 UTC)\r\n02:42:c0:a8:04:26 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:08.595059585 +0000 UTC)\r\n02:42:c0:a8:0f:07 -> 7a:b5:2d:25:d1:cf (2015-02-24 20:06:14.591152719 +0000 UTC)\r\n02:42:c0:a8:04:28 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.263309718 +0000 UTC)\r\n02:42:c0:a8:04:45 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.05489041 +0000 UTC)\r\n02:42:c0:a8:0e:05 -> 7a:8b:d8:d8:4a:88 (2015-02-24 20:06:16.222202156 +0000 UTC)\r\n02:42:c0:a8:10:15 -> 7a:8f:17:40:9e:ce (2015-02-24 20:06:29.391199438 +0000 UTC)\r\n7a:87:ef:4c:d7:5e -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:18.046407411 +0000 UTC)\r\n02:42:c0:a8:04:25 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:00:07.995434096 +0000 UTC)\r\n02:42:c0:a8:10:14 -> 7a:8f:17:40:9e:ce (2015-02-24 20:04:18.771543736 +0000 UTC)\r\n02:42:c0:a8:04:2a -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.369232214 +0000 UTC)\r\n02:42:c0:a8:04:2b -> 7a:87:ef:4c:d7:5e (2015-02-24 20:01:05.339005541 +0000 UTC)\r\n02:42:c0:a8:02:11 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:04:18.773508746 +0000 UTC)\r\n02:42:c0:a8:04:35 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:05:12.590450824 +0000 UTC)\r\n02:42:c0:a8:04:47 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:06:04.566932676 +0000 UTC)\r\n02:42:c0:a8:04:2c -> 7a:87:ef:4c:d7:5e (2015-02-24 20:02:04.785601517 +0000 UTC)\r\n02:42:c0:a8:02:01 -> 7a:8c:a1:e4:43:13 (2015-02-24 20:05:47.362256374 +0000 UTC)\r\n02:42:c0:a8:04:17 -> 7a:87:ef:4c:d7:5e (2015-02-24 19:57:03.296895233 +0000 UTC)\r\n02:42:c0:a8:04:30 -> 7a:87:ef:4c:d7:5e (2015-02-24 20:03:05.54918639 +0000 UTC)\r\nPeers:\r\nPeer 7a:87:ef:4c:d7:5e (v140) (UID 4530646544833339324)\r\n   -> 7a:8f:17:40:9e:ce [192.168.4.254:49022]\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.4.254:48716]\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:b5:2d:25:d1:cf [192.168.4.254:39140]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.4.254:35585]\r\nPeer 7a:8b:d8:d8:4a:88 (v894) (UID 3299225439164496535)\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:8c:a1:e4:43:13 [192.168.2.254:6783]\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\nPeer 7a:8f:17:40:9e:ce (v19) (UID 10167509669896236954)\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783]\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nPeer 7a:8c:a1:e4:43:13 (v1664) (UID 10303124246416533281)\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.2.254:60715 (unestablished)]\r\n   -> 7a:8f:17:40:9e:ce [192.168.2.254:51743]\r\n   -> 7a:87:ef:4c:d7:5e [192.168.2.254:43466]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.2.254:46923 (unestablished)]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:b5:2d:25:d1:cf [192.168.2.254:60685]\r\nPeer 7a:02:0f:ac:19:f4 (v902) (UID 6948683680276824273)\r\n   -> 7a:87:ef:4c:d7:5e [192.168.4.254:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\n   -> 7a:8c:a1:e4:43:13 [10.179.163.107:6783 (unestablished)]\r\nPeer 7a:21:54:68:1a:41 (v298) (UID 16892061667268854472)\r\n   -> 7a:87:ef:4c:d7:5e [192.168.1.254:53988]\r\n   -> 7a:b5:2d:25:d1:cf [192.168.1.254:51106]\r\n   -> 7a:8f:17:40:9e:ce [192.168.1.254:45757]\r\n   -> 7a:8c:a1:e4:43:13 [192.168.1.254:50783]\r\n   -> 7a:8b:d8:d8:4a:88 [192.168.1.254:48383]\r\n   -> 7a:02:0f:ac:19:f4 [192.168.1.254:37885]\r\nPeer 7a:b5:2d:25:d1:cf (v1432) (UID 2786178580184412291)\r\n   -> 7a:8c:a1:e4:43:13 [192.168.2.254:6783]\r\n   -> 7a:87:ef:4c:d7:5e [10.9.168.20:6783]\r\n   -> 7a:21:54:68:1a:41 [10.99.185.203:6783]\r\nRoutes:\r\nunicast:\r\n7a:8c:a1:e4:43:13 -> 00:00:00:00:00:00\r\n7a:21:54:68:1a:41 -> 7a:21:54:68:1a:41\r\n7a:b5:2d:25:d1:cf -> 7a:b5:2d:25:d1:cf\r\n7a:8f:17:40:9e:ce -> 7a:8f:17:40:9e:ce\r\n7a:87:ef:4c:d7:5e -> 7a:87:ef:4c:d7:5e\r\n7a:8b:d8:d8:4a:88 -> 7a:21:54:68:1a:41\r\n7a:02:0f:ac:19:f4 -> 7a:21:54:68:1a:41\r\nbroadcast:\r\n7a:87:ef:4c:d7:5e -> []\r\n7a:8b:d8:d8:4a:88 -> []\r\n7a:8f:17:40:9e:ce -> []\r\n7a:8c:a1:e4:43:13 -> [7a:02:0f:ac:19:f4 7a:21:54:68:1a:41 7a:b5:2d:25:d1:cf 7a:8b:d8:d8:4a:88 7a:8f:17:40:9e:ce 7a:87:ef:4c:d7:5e]\r\n7a:02:0f:ac:19:f4 -> []\r\n7a:21:54:68:1a:41 -> []\r\n7a:b5:2d:25:d1:cf -> []\r\nReconnects:\r\n10.9.168.20:6783 (next try at 2015-02-24 20:14:11.343347148 +0000 UTC)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/zettio/weave/issues/comments/76003225","html_url":"https://github.com/zettio/weave/issues/416#issuecomment-76003225","issue_url":"https://api.github.com/repos/zettio/weave/issues/416","id":76003225,"user":{"login":"bmurphy1976","id":270488,"avatar_url":"https://avatars.githubusercontent.com/u/270488?v=3","gravatar_id":"","url":"https://api.github.com/users/bmurphy1976","html_url":"https://github.com/bmurphy1976","followers_url":"https://api.github.com/users/bmurphy1976/followers","following_url":"https://api.github.com/users/bmurphy1976/following{/other_user}","gists_url":"https://api.github.com/users/bmurphy1976/gists{/gist_id}","starred_url":"https://api.github.com/users/bmurphy1976/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmurphy1976/subscriptions","organizations_url":"https://api.github.com/users/bmurphy1976/orgs","repos_url":"https://api.github.com/users/bmurphy1976/repos","events_url":"https://api.github.com/users/bmurphy1976/events{/privacy}","received_events_url":"https://api.github.com/users/bmurphy1976/received_events","type":"User","site_admin":false},"created_at":"2015-02-25T17:03:51Z","updated_at":"2015-02-25T17:03:51Z","body":"10.99.185.203 is server (A) in the comments above.\r\n\r\nHere's more logs, stack dumps, and weave status for the four servers above.  At the moment, X cannot connect to C but it can connect to A and B.\r\n\r\nX: https://gist.github.com/bmurphy1976/bb30257d4a33d98d2bf6\r\nA: https://gist.github.com/bmurphy1976/e1707e75902f836f28dc\r\nB: https://gist.github.com/bmurphy1976/a02dff988d7d0eccc76d\r\nC: https://gist.github.com/bmurphy1976/cf083e2be4781dd56b94"}}