{"action":"created","issue":{"url":"https://api.github.com/repos/rackerlabs/otter/issues/881","labels_url":"https://api.github.com/repos/rackerlabs/otter/issues/881/labels{/name}","comments_url":"https://api.github.com/repos/rackerlabs/otter/issues/881/comments","events_url":"https://api.github.com/repos/rackerlabs/otter/issues/881/events","html_url":"https://github.com/rackerlabs/otter/issues/881","id":53208098,"number":881,"title":"Remove deleted server's nodes from LBs","user":{"login":"manishtomar","id":758529,"avatar_url":"https://avatars.githubusercontent.com/u/758529?v=3","gravatar_id":"","url":"https://api.github.com/users/manishtomar","html_url":"https://github.com/manishtomar","followers_url":"https://api.github.com/users/manishtomar/followers","following_url":"https://api.github.com/users/manishtomar/following{/other_user}","gists_url":"https://api.github.com/users/manishtomar/gists{/gist_id}","starred_url":"https://api.github.com/users/manishtomar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manishtomar/subscriptions","organizations_url":"https://api.github.com/users/manishtomar/orgs","repos_url":"https://api.github.com/users/manishtomar/repos","events_url":"https://api.github.com/users/manishtomar/events{/privacy}","received_events_url":"https://api.github.com/users/manishtomar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rackerlabs/otter/milestones/5","html_url":"https://github.com/rackerlabs/otter/milestones/Convergence:%20CLB","labels_url":"https://api.github.com/repos/rackerlabs/otter/milestones/5/labels","id":974143,"number":5,"title":"Convergence: CLB","description":"Get to the point that we can allow CLB convergence for some tenants who make use of CLB.","creator":{"login":"glyph","id":716529,"avatar_url":"https://avatars.githubusercontent.com/u/716529?v=3","gravatar_id":"","url":"https://api.github.com/users/glyph","html_url":"https://github.com/glyph","followers_url":"https://api.github.com/users/glyph/followers","following_url":"https://api.github.com/users/glyph/following{/other_user}","gists_url":"https://api.github.com/users/glyph/gists{/gist_id}","starred_url":"https://api.github.com/users/glyph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glyph/subscriptions","organizations_url":"https://api.github.com/users/glyph/orgs","repos_url":"https://api.github.com/users/glyph/repos","events_url":"https://api.github.com/users/glyph/events{/privacy}","received_events_url":"https://api.github.com/users/glyph/received_events","type":"User","site_admin":false},"open_issues":13,"closed_issues":5,"state":"open","created_at":"2015-02-10T19:04:49Z","updated_at":"2015-05-12T22:55:20Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2014-12-31T23:26:06Z","updated_at":"2015-05-15T22:19:46Z","closed_at":null,"body":"Convergence should remove any out-of-band deleted servers from all the LBs it was attached to. This requires getting deleted servers from `get_scaling_group_servers` which needs time since it was last fetched. [Docs](http://docs.rackspace.com/servers/api/v2/cs-devguide/content/ChangesSince.html) for this. Otherwise Nova will return all servers deleted in last 30 days. It is not clear where this state will be stored. It would most likely be otter db. So, this task can be divided in:\r\n- [ ] Decide where to store last fetched info. The info includes servers fetched and time they were fetched\r\n   - [ ] If it is otter db, then define model interface and implement it\r\n- [ ] Add `changes_since` support in mimic\r\n- [x] Add support to get deleted servers in `get_server_details` and `get_scaling_group_servers`: #1252\r\n- [x] Change `converge` to remove dangling LB nodes"},"comment":{"url":"https://api.github.com/repos/rackerlabs/otter/issues/comments/102537953","html_url":"https://github.com/rackerlabs/otter/issues/881#issuecomment-102537953","issue_url":"https://api.github.com/repos/rackerlabs/otter/issues/881","id":102537953,"user":{"login":"cyli","id":223200,"avatar_url":"https://avatars.githubusercontent.com/u/223200?v=3","gravatar_id":"","url":"https://api.github.com/users/cyli","html_url":"https://github.com/cyli","followers_url":"https://api.github.com/users/cyli/followers","following_url":"https://api.github.com/users/cyli/following{/other_user}","gists_url":"https://api.github.com/users/cyli/gists{/gist_id}","starred_url":"https://api.github.com/users/cyli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyli/subscriptions","organizations_url":"https://api.github.com/users/cyli/orgs","repos_url":"https://api.github.com/users/cyli/repos","events_url":"https://api.github.com/users/cyli/events{/privacy}","received_events_url":"https://api.github.com/users/cyli/received_events","type":"User","site_admin":false},"created_at":"2015-05-15T22:19:46Z","updated_at":"2015-05-15T22:19:46Z","body":"@manishtomar and @wirehead and I were discussing this today, and a couple of things came out of the discussion:\r\n\r\n1.  @manishtomar suggested a solution that means we don't have to do any fancy figuring out of which servers have been cleaned up and which have yet to be cleaned up:\r\n    1.  A convergence cycle starts, and we gather any deleted servers since time T=0.  The gathering happens at time T=1.\r\n    2.  If any of those servers are still attached to any load balancers, this convergence cycle will attempt to remove them.\r\n    3.  Hence if the convergence cycle succeeds at time T=2, that means that the load balancer nodes were all removed successfully (or that the server was not on any load balancers to begin with), and so we can record T=1 the last fetched time.\r\n    4.  If the convergence cycle needs to retry for any reason, we do not change the last fetched time.  (It doesn't matter whether it failed for some other reason - it just means we check again next time.)  If the convergence cycle fails, we again do not change the last fetched time.\r\n\r\n2.  We discussed a heuristic we use to be more confident that the IP was not re-used, even if the server was deleted 30 days ago.\r\n    1. When we see a deleted server, we check the time it was deleted (probably the last changed date in the server details JSON).\r\n    2. If we discover its IP is on a CLB, we check the create time of the corresponding node.  If it was at or before the server delete, we can be reasonably sure the node was created for this server, and clean it up.  We can even remove the node if the node creation time was the server deletion time, so long as the time difference is small (way before the IP recycle period).  Otherwise, we don't touch the node.\r\n\r\n3.  @wirehead suggests that the recycle period for servicenet IPs (which is all we care about for CLBs) is pretty long, since there are a lot of servicenet IPs.\r\n"}}