{"action":"closed","issue":{"url":"https://api.github.com/repos/Metaswitch/chef/issues/179","labels_url":"https://api.github.com/repos/Metaswitch/chef/issues/179/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/chef/issues/179/comments","events_url":"https://api.github.com/repos/Metaswitch/chef/issues/179/events","html_url":"https://github.com/Metaswitch/chef/issues/179","id":80282491,"number":179,"title":"Can't scale down large EC2 deployment due to throttling","user":{"login":"mirw","id":2963879,"avatar_url":"https://avatars.githubusercontent.com/u/2963879?v=3","gravatar_id":"","url":"https://api.github.com/users/mirw","html_url":"https://github.com/mirw","followers_url":"https://api.github.com/users/mirw/followers","following_url":"https://api.github.com/users/mirw/following{/other_user}","gists_url":"https://api.github.com/users/mirw/gists{/gist_id}","starred_url":"https://api.github.com/users/mirw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirw/subscriptions","organizations_url":"https://api.github.com/users/mirw/orgs","repos_url":"https://api.github.com/users/mirw/repos","events_url":"https://api.github.com/users/mirw/events{/privacy}","received_events_url":"https://api.github.com/users/mirw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/chef/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/chef/labels/medium-priority","name":"medium-priority","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"johadalin","id":14074163,"avatar_url":"https://avatars.githubusercontent.com/u/14074163?v=3","gravatar_id":"","url":"https://api.github.com/users/johadalin","html_url":"https://github.com/johadalin","followers_url":"https://api.github.com/users/johadalin/followers","following_url":"https://api.github.com/users/johadalin/following{/other_user}","gists_url":"https://api.github.com/users/johadalin/gists{/gist_id}","starred_url":"https://api.github.com/users/johadalin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johadalin/subscriptions","organizations_url":"https://api.github.com/users/johadalin/orgs","repos_url":"https://api.github.com/users/johadalin/repos","events_url":"https://api.github.com/users/johadalin/events{/privacy}","received_events_url":"https://api.github.com/users/johadalin/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-05-24T21:55:44Z","updated_at":"2015-09-29T16:08:48Z","closed_at":"2015-09-29T16:08:48Z","body":"Scale down a large (more than about 20 nodes) deployment on EC2.  This fails while updating DNS because \"Rate exceeded\".\r\n\r\nThis is because, when preparing to quiesce, we call KnifeDnsRecordsCreate, which spins through every DNS record (even nodes that are staying) and updates them in EC2.  This generates so much load (and has no pacing) that EC2 rejects it.\r\n\r\nThe fix is probably to pace our requests better, but alternatively, we might not need to do full KnifeDnsRecordsCreate processing, so we might be able to simplify there too."}}