{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/3291","labels_url":"https://api.github.com/repos/deis/deis/issues/3291/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/3291/comments","events_url":"https://api.github.com/repos/deis/deis/issues/3291/events","html_url":"https://github.com/deis/deis/issues/3291","id":61068126,"number":3291,"title":"login and register hangs","user":{"login":"ineu","id":24980,"avatar_url":"https://avatars.githubusercontent.com/u/24980?v=3","gravatar_id":"","url":"https://api.github.com/users/ineu","html_url":"https://github.com/ineu","followers_url":"https://api.github.com/users/ineu/followers","following_url":"https://api.github.com/users/ineu/following{/other_user}","gists_url":"https://api.github.com/users/ineu/gists{/gist_id}","starred_url":"https://api.github.com/users/ineu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ineu/subscriptions","organizations_url":"https://api.github.com/users/ineu/orgs","repos_url":"https://api.github.com/users/ineu/repos","events_url":"https://api.github.com/users/ineu/events{/privacy}","received_events_url":"https://api.github.com/users/ineu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-03-13T13:52:03Z","updated_at":"2015-03-27T19:07:24Z","closed_at":null,"body":"Hi guys.\r\nI've built new cluster of 3 bare metal machines living in Hetzner. CoreOS beta (607), Deis 1.4.1.\r\nImmediately after setup I was able to register, but few hours later both `deis register` and `deis login` hangs after entering all values. If I interrupt it with `C-c`, I see\r\n```\r\n^CTraceback (most recent call last):\r\n  File \"<string>\", line 2113, in <module>\r\n  File \"<string>\", line 2109, in main\r\n  File \"<string>\", line 2052, in _dispatch_cmd\r\n  File \"<string>\", line 801, in auth_login\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.sessions\", line 504, in post\r\n  File \"<string>\", line 158, in request\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.sessions\", line 461, in request\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.sessions\", line 573, in send\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.adapters\", line 370, in send\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.packages.urllib3.connectionpool\", line 518, in urlopen\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/requests.packages.urllib3.connectionpool\", line 353, in _make_request\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/httplib\", line 1045, in getresponse\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/httplib\", line 409, in begin\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/httplib\", line 365, in _read_status\r\n  File \"/home/jenkins/workspace/build-deis-cli-installer-linux/src/github.com/deis/deis/client/build/deis/out00-PYZ.pyz/socket\", line 476, in readline\r\nKeyboardInterrupt\r\n```\r\nI've checked etcd logs on all servers, two are fine, one emits errors:\r\n```\r\nMar 13 13:43:24 staging-coreos-2 etcd[625]: [etcd] Mar 13 13:43:24.551 INFO      | a6efd505a1c5420f8ab1c11af55571b3: warning: heartbeat time out peer=\"2fdc33aa2bb347aaa1da0d742d41190c\" missed=1 backoff=\"2s\"\r\nMar 13 13:44:08 staging-coreos-2 etcd[625]: [etcd] Mar 13 13:44:08.551 INFO      | a6efd505a1c5420f8ab1c11af55571b3: warning: heartbeat time out peer=\"2fdc33aa2bb347aaa1da0d742d41190c\" missed=1 backoff=\"2s\"\r\nMar 13 13:44:24 staging-coreos-2 etcd[625]: [etcd] Mar 13 13:44:24.761 INFO      | a6efd505a1c5420f8ab1c11af55571b3: warning: heartbeat time out peer=\"2fdc33aa2bb347aaa1da0d742d41190c\" missed=1 backoff=\"2s\"\r\nMar 13 13:44:34 staging-coreos-2 etcd[625]: [etcd] Mar 13 13:44:34.051 INFO      | a6efd505a1c5420f8ab1c11af55571b3: warning: heartbeat time out peer=\"2fdc33aa2bb347aaa1da0d742d41190c\" missed=1 backoff=\"2s\"\r\n```\r\nTime is the same on all three servers (up to seconds), though `ceph -s` reports\r\n```\r\n     health HEALTH_WARN clock skew detected on mon.staging-coreos-2, mon.staging-coreos-3\r\n```\r\nThough maybe ceph requires more precise time. Ntpd is running on all servers.\r\nWhat can I do to further investigate the issue? Thank you.\r\n"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/87059983","html_url":"https://github.com/deis/deis/issues/3291#issuecomment-87059983","issue_url":"https://api.github.com/repos/deis/deis/issues/3291","id":87059983,"user":{"login":"gabrtv","id":231123,"avatar_url":"https://avatars.githubusercontent.com/u/231123?v=3","gravatar_id":"","url":"https://api.github.com/users/gabrtv","html_url":"https://github.com/gabrtv","followers_url":"https://api.github.com/users/gabrtv/followers","following_url":"https://api.github.com/users/gabrtv/following{/other_user}","gists_url":"https://api.github.com/users/gabrtv/gists{/gist_id}","starred_url":"https://api.github.com/users/gabrtv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gabrtv/subscriptions","organizations_url":"https://api.github.com/users/gabrtv/orgs","repos_url":"https://api.github.com/users/gabrtv/repos","events_url":"https://api.github.com/users/gabrtv/events{/privacy}","received_events_url":"https://api.github.com/users/gabrtv/received_events","type":"User","site_admin":false},"created_at":"2015-03-27T19:07:24Z","updated_at":"2015-03-27T19:07:24Z","body":"@ineu thanks for the detailed debug output.  Extremely helpful.  \r\n\r\nI have a strong suspicion the issue is caused by the database publishing logic.  The `/deis/database/host` key is written with a TTL inside a loop that can potentially be interrupted by periodic backups.  If backups take longer than the TTL, the keys can disappear from etcd.  I suspect that's what is happening here.\r\n\r\nTo resolve this, we need to decouple _publishing_ from _period backups_.  The best way to do this is to rewrite the database init in Go and use goroutines for background tasks, as we do elsewhere."}}