{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/1593","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/1593/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/1593/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/1593/events","html_url":"https://github.com/weaveworks/weave/issues/1593","id":113545984,"number":1593,"title":"weave script hangs in IP address allocation after rolling restart","user":{"login":"dpw","id":15074,"avatar_url":"https://avatars.githubusercontent.com/u/15074?v=3","gravatar_id":"","url":"https://api.github.com/users/dpw","html_url":"https://github.com/dpw","followers_url":"https://api.github.com/users/dpw/followers","following_url":"https://api.github.com/users/dpw/following{/other_user}","gists_url":"https://api.github.com/users/dpw/gists{/gist_id}","starred_url":"https://api.github.com/users/dpw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpw/subscriptions","organizations_url":"https://api.github.com/users/dpw/orgs","repos_url":"https://api.github.com/users/dpw/repos","events_url":"https://api.github.com/users/dpw/events{/privacy}","received_events_url":"https://api.github.com/users/dpw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/weaveworks/weave/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/weaveworks/weave/labels/[component/ipam]","name":"[component/ipam]","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-27T09:53:16Z","updated_at":"2015-10-28T14:24:23Z","closed_at":null,"body":"With 1.2.0, if I:\r\n\r\n* Start two peers (one just with `weave launch`, one with `weave launch <other's ip>`\r\n* Run a short-lived container on each host in turn: `docker $(weave config) run --name=foo --rm ubuntu ls` (naming the containers seems to be necessary).\r\n* On each host in turn, do `weave reset` and `weave launch` (the latter with the original args).  I.e. at least one router is running throughout.\r\n\r\nthen further attempt to start containers on the weave network hang (e.g. `weave run -itd ubuntu` on either host).  Seems to be 100% reproducible.  Last few lines with WEAVE_DEBUG=1:\r\n\r\n```\r\n+ IPAM_URL=/ip/ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b\r\n+ call_weave POST /ip/ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b?check-alive=true\r\n+ container_ip weave weave container is not present. Have you launched it? weave container is not running.\r\n+ docker inspect --format={{.State.Running}} {{.NetworkSettings.IPAddress}} weave\r\n+ status=true \r\n+ CONTAINER_IP=127.0.0.1\r\n+ http_call 127.0.0.1:6784 POST /ip/ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b?check-alive=true\r\n+ addr=127.0.0.1:6784\r\n+ http_verb=POST\r\n+ url=/ip/ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b?check-alive=true\r\n+ shift 3\r\n+ curl --connect-timeout 3 -s -S -X POST http://127.0.0.1:6784/ip/ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b?check-alive=true\r\n```\r\n\r\nIt's not caused by IPAM consensus woes.  `weave status`:\r\n\r\n```\r\n...\r\n       Service: ipam\r\n     Consensus: achieved\r\n         Range: [10.32.0.0-10.48.0.0)\r\n DefaultSubnet: 10.32.0.0/12\r\n...\r\n```\r\n\r\nNothing obviously interesting in the weave logs.  Last few lines are\r\n\r\n```\r\nINFO: 2015/10/27 09:38:39.678852 [allocator]: Allocate request for 973fd8e60a84dc00aa83bef5c4e7c71015b5229c3d9c564586f7e9955699bb57 cancelled\r\nINFO: 2015/10/27 09:41:22.981950 [allocator]: Allocate request for 2ca43780f78dc2809b20ab79401f2d5f7257d37ba0d28b6d3b1ca2e7a0d297fc cancelled\r\nINFO: 2015/10/27 09:42:52.937975 [allocator]: Allocate request for cbb3c83b9b705e327f948d104bcefc5faa954919b408b9e7258452b9c13c43fe cancelled\r\nINFO: 2015/10/27 09:44:52.927980 [allocator]: Allocate request for ab7015b9774d10f7eb365c9f5aa0264d5e6887dc3fb0dc12bc31944d87aefa8b cancelled\r\n```\r\n\r\nDue to me killing the hung weave scripts.  So it is getting the HTTP requests, and getting stuck while processing them.\r\n\r\nI have my hands full with other stuff today, so I'm not planning to look further into this right now.  Might be good if someone else confirms."},"comment":{"url":"https://api.github.com/repos/weaveworks/weave/issues/comments/151860741","html_url":"https://github.com/weaveworks/weave/issues/1593#issuecomment-151860741","issue_url":"https://api.github.com/repos/weaveworks/weave/issues/1593","id":151860741,"user":{"login":"dpw","id":15074,"avatar_url":"https://avatars.githubusercontent.com/u/15074?v=3","gravatar_id":"","url":"https://api.github.com/users/dpw","html_url":"https://github.com/dpw","followers_url":"https://api.github.com/users/dpw/followers","following_url":"https://api.github.com/users/dpw/following{/other_user}","gists_url":"https://api.github.com/users/dpw/gists{/gist_id}","starred_url":"https://api.github.com/users/dpw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpw/subscriptions","organizations_url":"https://api.github.com/users/dpw/orgs","repos_url":"https://api.github.com/users/dpw/repos","events_url":"https://api.github.com/users/dpw/events{/privacy}","received_events_url":"https://api.github.com/users/dpw/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T14:24:23Z","updated_at":"2015-10-28T14:24:23Z","body":"> I doubt this is new in 1.2. @tomwilkie\r\n\r\nIndeed, I can reproduce it in 1.1.2.\r\n\r\nDoesn't this somewhat undermine our support for rolling upgrades?"}}