{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/4731","labels_url":"https://api.github.com/repos/deis/deis/issues/4731/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/4731/comments","events_url":"https://api.github.com/repos/deis/deis/issues/4731/events","html_url":"https://github.com/deis/deis/issues/4731","id":115688627,"number":4731,"title":"Controller boot takes hours while dumping the database to etcd","user":{"login":"jwaldrip","id":43164,"avatar_url":"https://avatars.githubusercontent.com/u/43164?v=3","gravatar_id":"","url":"https://api.github.com/users/jwaldrip","html_url":"https://github.com/jwaldrip","followers_url":"https://api.github.com/users/jwaldrip/followers","following_url":"https://api.github.com/users/jwaldrip/following{/other_user}","gists_url":"https://api.github.com/users/jwaldrip/gists{/gist_id}","starred_url":"https://api.github.com/users/jwaldrip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwaldrip/subscriptions","organizations_url":"https://api.github.com/users/jwaldrip/orgs","repos_url":"https://api.github.com/users/jwaldrip/repos","events_url":"https://api.github.com/users/jwaldrip/events{/privacy}","received_events_url":"https://api.github.com/users/jwaldrip/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/database","name":"database","color":"bfd4f2"},{"url":"https://api.github.com/repos/deis/deis/labels/store%20(Ceph)","name":"store (Ceph)","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-11-07T20:01:10Z","updated_at":"2015-12-03T13:37:33Z","closed_at":null,"body":"I am upgrading to the latest deis, when starting the controller, it spends almost 3 hours dumping its settings to etcd.\r\n\r\nIts some of these but for every config record in the db. Each one takes 30 seconds to a minute. I have 117 in the DB, so this is taking a while... please don't tell me this happens every time it boots...\r\n\r\nSample Line from log: \r\n```\r\nINFO boulder-production: config boulder-production-4621fa0 updated\r\n```"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/161641919","html_url":"https://github.com/deis/deis/issues/4731#issuecomment-161641919","issue_url":"https://api.github.com/repos/deis/deis/issues/4731","id":161641919,"user":{"login":"ineu","id":24980,"avatar_url":"https://avatars.githubusercontent.com/u/24980?v=3","gravatar_id":"","url":"https://api.github.com/users/ineu","html_url":"https://github.com/ineu","followers_url":"https://api.github.com/users/ineu/followers","following_url":"https://api.github.com/users/ineu/following{/other_user}","gists_url":"https://api.github.com/users/ineu/gists{/gist_id}","starred_url":"https://api.github.com/users/ineu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ineu/subscriptions","organizations_url":"https://api.github.com/users/ineu/orgs","repos_url":"https://api.github.com/users/ineu/repos","events_url":"https://api.github.com/users/ineu/events{/privacy}","received_events_url":"https://api.github.com/users/ineu/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T13:37:33Z","updated_at":"2015-12-03T13:37:33Z","body":"I was too optimistic. Upgraded to 1.12.2, controller still takes very long to start:\r\n```\r\ncore@staging-2 ~ $ journalctl -b -u deis-controller | grep 'updated$' | wc -l\r\n96\r\n```\r\n\r\nIt took 9 minutes to execute all these updates, all this time builder was waiting for controller's data in etcd."}}