{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/3300","labels_url":"https://api.github.com/repos/deis/deis/issues/3300/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/3300/comments","events_url":"https://api.github.com/repos/deis/deis/issues/3300/events","html_url":"https://github.com/deis/deis/pull/3300","id":61708007,"number":3300,"title":"feat(cloud-init): temporal fix for ntpd error and clock skew","user":{"login":"aledbf","id":161571,"avatar_url":"https://avatars.githubusercontent.com/u/161571?v=3","gravatar_id":"","url":"https://api.github.com/users/aledbf","html_url":"https://github.com/aledbf","followers_url":"https://api.github.com/users/aledbf/followers","following_url":"https://api.github.com/users/aledbf/following{/other_user}","gists_url":"https://api.github.com/users/aledbf/gists{/gist_id}","starred_url":"https://api.github.com/users/aledbf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aledbf/subscriptions","organizations_url":"https://api.github.com/users/aledbf/orgs","repos_url":"https://api.github.com/users/aledbf/repos","events_url":"https://api.github.com/users/aledbf/events{/privacy}","received_events_url":"https://api.github.com/users/aledbf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/contrib%20and%20provisioning","name":"contrib and provisioning","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"aledbf","id":161571,"avatar_url":"https://avatars.githubusercontent.com/u/161571?v=3","gravatar_id":"","url":"https://api.github.com/users/aledbf","html_url":"https://github.com/aledbf","followers_url":"https://api.github.com/users/aledbf/followers","following_url":"https://api.github.com/users/aledbf/following{/other_user}","gists_url":"https://api.github.com/users/aledbf/gists{/gist_id}","starred_url":"https://api.github.com/users/aledbf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aledbf/subscriptions","organizations_url":"https://api.github.com/users/aledbf/orgs","repos_url":"https://api.github.com/users/aledbf/repos","events_url":"https://api.github.com/users/aledbf/events{/privacy}","received_events_url":"https://api.github.com/users/aledbf/received_events","type":"User","site_admin":false},"milestone":null,"comments":13,"created_at":"2015-03-14T22:45:03Z","updated_at":"2015-03-19T01:38:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/deis/deis/pulls/3300","html_url":"https://github.com/deis/deis/pull/3300","diff_url":"https://github.com/deis/deis/pull/3300.diff","patch_url":"https://github.com/deis/deis/pull/3300.patch"},"body":"replaces #3222 \r\n\r\nReferences:\r\n- https://github.com/coreos/bugs/issues/304\r\n- https://github.com/coreos/coreos-overlay/pull/1145\r\n\r\n**This should be temporal until the overlay PR is available in CoreOS stable**"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/83253016","html_url":"https://github.com/deis/deis/pull/3300#issuecomment-83253016","issue_url":"https://api.github.com/repos/deis/deis/issues/3300","id":83253016,"user":{"login":"ianblenke","id":1844764,"avatar_url":"https://avatars.githubusercontent.com/u/1844764?v=3","gravatar_id":"","url":"https://api.github.com/users/ianblenke","html_url":"https://github.com/ianblenke","followers_url":"https://api.github.com/users/ianblenke/followers","following_url":"https://api.github.com/users/ianblenke/following{/other_user}","gists_url":"https://api.github.com/users/ianblenke/gists{/gist_id}","starred_url":"https://api.github.com/users/ianblenke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianblenke/subscriptions","organizations_url":"https://api.github.com/users/ianblenke/orgs","repos_url":"https://api.github.com/users/ianblenke/repos","events_url":"https://api.github.com/users/ianblenke/events{/privacy}","received_events_url":"https://api.github.com/users/ianblenke/received_events","type":"User","site_admin":false},"created_at":"2015-03-19T01:38:24Z","updated_at":"2015-03-19T01:38:24Z","body":"Also, to @carmstrong's question: we were running HVM and seeing this problem. Since then, there have been folks who have presented problems with PV as well (an 8 second drift on one node report in the past couple of days). When a problem is encountered with PV clock drift, my current thinking is that the underlying Xen host is having NTP issues, and PV guests inherit this clock drift through the hypervisor as a result.\r\n\r\nRunning ntp inside a VM with an independent wallclock is asking for drift, as jiffies get stolen by other bursty bad actors on the same Xen host.\r\n\r\nTrusting the underlying Xen host will always have correct time with no drift seems to have been proven wrong at least occasionally as well. The two solutions here are:\r\n\r\n1. To enable an independent wallclock on PV and run ntpd (which conflicts with the point above), which we know to be problematic with HVM today.\r\n\r\n2. Shutdown/stop the PV instance and start it back up, so that the instance gets moved to another Xen host in the AWS farm that potentially has the correct NTP time.\r\n\r\n3. Fiddle with `mon-clock-drift-allowed` with the guidance of the Ceph project and see if we can't allow a little more wiggle room than 0.05 seconds for ntp and stolen jiffies in a VM:\r\n\r\nhttp://ceph.com/docs/master/rados/troubleshooting/troubleshooting-mon/#clock-skews\r\n\r\n"}}