{"action":"created","issue":{"url":"https://api.github.com/repos/zettio/weave/issues/367","labels_url":"https://api.github.com/repos/zettio/weave/issues/367/labels{/name}","comments_url":"https://api.github.com/repos/zettio/weave/issues/367/comments","events_url":"https://api.github.com/repos/zettio/weave/issues/367/events","html_url":"https://github.com/zettio/weave/issues/367","id":55487425,"number":367,"title":"Internal GC can leave Peers in inconsistent state","user":{"login":"bboreham","id":8125524,"avatar_url":"https://avatars.githubusercontent.com/u/8125524?v=3","gravatar_id":"","url":"https://api.github.com/users/bboreham","html_url":"https://github.com/bboreham","followers_url":"https://api.github.com/users/bboreham/followers","following_url":"https://api.github.com/users/bboreham/following{/other_user}","gists_url":"https://api.github.com/users/bboreham/gists{/gist_id}","starred_url":"https://api.github.com/users/bboreham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bboreham/subscriptions","organizations_url":"https://api.github.com/users/bboreham/orgs","repos_url":"https://api.github.com/users/bboreham/repos","events_url":"https://api.github.com/users/bboreham/events{/privacy}","received_events_url":"https://api.github.com/users/bboreham/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zettio/weave/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-01-26T14:23:41Z","updated_at":"2015-01-31T18:21:17Z","closed_at":null,"body":"Suppose we have three peers with three directed connections as follows:\r\n\r\n1 <--> 2 --> 3\r\n\r\nThen when node 3 does a GarbageCollect, it will remove node 1 as unreachable, while still retaining a reference to node 1 as a connection from node 2.  Node 2 is retained because it is 'locally referenced', i.e. there is a connection with 3.\r\n\r\nIf node 3 now tries to send this topology to another node that doesn't know 1, the update will be rejected as it mentions but does not contain node 1."},"comment":{"url":"https://api.github.com/repos/zettio/weave/issues/comments/72329219","html_url":"https://github.com/zettio/weave/issues/367#issuecomment-72329219","issue_url":"https://api.github.com/repos/zettio/weave/issues/367","id":72329219,"user":{"login":"rade","id":109109,"avatar_url":"https://avatars.githubusercontent.com/u/109109?v=3","gravatar_id":"","url":"https://api.github.com/users/rade","html_url":"https://github.com/rade","followers_url":"https://api.github.com/users/rade/followers","following_url":"https://api.github.com/users/rade/following{/other_user}","gists_url":"https://api.github.com/users/rade/gists{/gist_id}","starred_url":"https://api.github.com/users/rade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rade/subscriptions","organizations_url":"https://api.github.com/users/rade/orgs","repos_url":"https://api.github.com/users/rade/repos","events_url":"https://api.github.com/users/rade/events{/privacy}","received_events_url":"https://api.github.com/users/rade/received_events","type":"User","site_admin":false},"created_at":"2015-01-31T18:21:17Z","updated_at":"2015-01-31T18:21:17Z","body":"Steps to reproduce the inconsistency...\r\n\r\n#### Step 1: patch & rebuild weave\r\n```patch\r\ndiff --git a/router/gossip.go b/router/gossip.go\r\nindex 5c37422..8596b7c 100644\r\n--- a/router/gossip.go\r\n+++ b/router/gossip.go\r\n@@ -8,7 +8,7 @@ import (\r\n \t\"time\"\r\n )\r\n \r\n-const GossipInterval = 30 * time.Second\r\n+const GossipInterval = 3 * time.Second\r\n \r\n type Gossip interface {\r\n \t// specific message from one peer to another\r\ndiff --git a/router/local_peer.go b/router/local_peer.go\r\nindex 0cbb3fd..418a46b 100644\r\n--- a/router/local_peer.go\r\n+++ b/router/local_peer.go\r\n@@ -7,6 +7,8 @@ import (\r\n \t\"time\"\r\n )\r\n \r\n+var LocalPeerDelay = 0\r\n+\r\n type LocalPeer struct {\r\n \t*Peer\r\n \tRouter    *Router\r\n@@ -166,6 +168,9 @@ func (peer *LocalPeer) queryLoop(queryChan <-chan *PeerInteraction) {\r\n \t\t\t}\r\n \t\t\tswitch query.code {\r\n \t\t\tcase PAddConnection:\r\n+\t\t\t\tif LocalPeerDelay > 0 {\r\n+\t\t\t\t\ttime.Sleep(time.Duration(LocalPeerDelay) * time.Second)\r\n+\t\t\t\t}\r\n \t\t\t\tconn := query.payload.(*LocalConnection)\r\n \t\t\t\tif peer.handleAddConnection(conn) {\r\n \t\t\t\t\tconn.log(\"connection added\")\r\ndiff --git a/weaver/main.go b/weaver/main.go\r\nindex de62f30..37916e5 100644\r\n--- a/weaver/main.go\r\n+++ b/weaver/main.go\r\n@@ -55,6 +55,7 @@ func main() {\r\n \tflag.StringVar(&prof, \"profile\", \"\", \"enable profiling and write profiles to given path\")\r\n \tflag.IntVar(&connLimit, \"connlimit\", 10, \"connection limit (defaults to 10, set to 0 for unlimited)\")\r\n \tflag.IntVar(&bufSz, \"bufsz\", 8, \"capture buffer size in MB (defaults to 8MB)\")\r\n+\tflag.IntVar(&weave.LocalPeerDelay, \"delay\", 0, \"local peer processing delay (for testing, defaults to 0)\")\r\n \tflag.Parse()\r\n \tpeers = flag.Args()\r\n```\r\nThis reduces the gossip interval (so we don't have to wait so long) and introduces a command line option to delay processing of `AddConnection` by `LocalPeer`.\r\n\r\n#### Step 2: fire up 4 VMs containing weave\r\nAs of the https://github.com/zettio/weave/commit/5967d5de206321e896af82713eeb93bd6e1c98f4 it is easy to spin up an arbitrary number of VMs containing weave.\r\n\r\n#### Step 3: start the weaves\r\n...and construct an initial topology of two pairs\r\n```\r\nhost1# weave launch\r\nhost2# weave launch host1-ip\r\nhost3# weave launch --delay=5\r\nhost4# weave launch host3-ip\r\n```\r\nCheck the `weave status` on 3 and 4 until the connection between them is fully established in both directions\r\n\r\n#### Step 4: connect the two pairs\r\n```\r\nhost3# weave connect host2-ip\r\n```\r\n\r\n#### Step 5: check the logs on 4\r\nIn the presence of the issue the logs contain \"Topology gossip: Reference to unknown peer ...\" entries.\r\n"}}