{"action":"created","issue":{"url":"https://api.github.com/repos/mesos/chronos/issues/508","labels_url":"https://api.github.com/repos/mesos/chronos/issues/508/labels{/name}","comments_url":"https://api.github.com/repos/mesos/chronos/issues/508/comments","events_url":"https://api.github.com/repos/mesos/chronos/issues/508/events","html_url":"https://github.com/mesos/chronos/issues/508","id":99322374,"number":508,"title":"Jobs may not run on schedule","user":{"login":"aphyr","id":3748,"avatar_url":"https://avatars.githubusercontent.com/u/3748?v=3","gravatar_id":"","url":"https://api.github.com/users/aphyr","html_url":"https://github.com/aphyr","followers_url":"https://api.github.com/users/aphyr/followers","following_url":"https://api.github.com/users/aphyr/following{/other_user}","gists_url":"https://api.github.com/users/aphyr/gists{/gist_id}","starred_url":"https://api.github.com/users/aphyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aphyr/subscriptions","organizations_url":"https://api.github.com/users/aphyr/orgs","repos_url":"https://api.github.com/users/aphyr/repos","events_url":"https://api.github.com/users/aphyr/events{/privacy}","received_events_url":"https://api.github.com/users/aphyr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-08-05T23:42:37Z","updated_at":"2015-08-06T22:57:09Z","closed_at":null,"body":"I'm testing Chronos with Jepsen--we'd like guarantees that jobs will be completed on schedule, even in the presence of network and node failures. I've had a spot of trouble getting started: jobs aren't invoked on schedule even *before* I've started inducing failure.\r\n\r\nThis test creates a five-node cluster: (n1, n2, n3, n4, n5). ZK runs on all five nodes, Mesos (0.23.0-1.0.debian81) masters run on the first three nodes (n1, n2, n3), and Mesos slaves run on n4 and n5. Chronos (2.3.4-1.0.81.debian77) runs on all five nodes.\r\n\r\nWe [generate and insert a handful of jobs into Chronos](https://github.com/aphyr/jepsen/blob/master/chronos/src/jepsen/chronos.clj) for 300 seconds, then wait 120 seconds to watch the system in a steady state, and finally read the results. Each job consists of a small shell script which allocates a tmpfile, logs the job name and start time, sleeps for `:duration` seconds, then logs the completion time. To read the final state, we take the union of all files written by every job. When a run writes its final time we consider it `complete`--if it only logs the initial time we consider it `incomplete`.\r\n\r\nJob schedules are selected such that the `:interval` between jobs is always greater than their `:duration` plus `:epsilon` plus an additional `epsilon-forgiveness`--in testing, it looks like Chronos will regularly slip deadlines by a second or two, so we allow an extra 5 seconds slop. This should guarantee that jobs never run concurrently.\r\n\r\nAll node clocks are perfectly synchronized.\r\n\r\nTo [verify correctness](https://github.com/aphyr/jepsen/blob/master/chronos/src/jepsen/chronos/checker.clj), we expand the job spec into a sequence of `:target` intervals: a vector of `[start-time end-time]` during which the job should have begun. We only compute targets which would have been complete at the time of the final read. Then, we take all completed invocations, and attempt to find a unique `:solution` mapping targets to runs, using the Choco constraint solver. A job is considered invalid if some target could not be satisfied by a completed run. For example:\r\n\r\n```clj\r\n{:valid? false,\r\n   :job\r\n   {:name 36,\r\n    :start #<DateTime 2015-08-05T21:57:42.512Z>,\r\n    :count 5,\r\n    :duration 6,\r\n    :epsilon 6,\r\n    :interval 24},\r\n   :solution nil,\r\n   :targets\r\n   ([#<DateTime 2015-08-05T21:57:42.512Z>\r\n     #<DateTime 2015-08-05T21:57:53.512Z>]\r\n    [#<DateTime 2015-08-05T21:58:06.512Z>\r\n     #<DateTime 2015-08-05T21:58:17.512Z>]\r\n    [#<DateTime 2015-08-05T21:58:30.512Z>\r\n     #<DateTime 2015-08-05T21:58:41.512Z>]\r\n    [#<DateTime 2015-08-05T21:58:54.512Z>\r\n     #<DateTime 2015-08-05T21:59:05.512Z>]\r\n    [#<DateTime 2015-08-05T21:59:18.512Z>\r\n     #<DateTime 2015-08-05T21:59:29.512Z>]),\r\n   :extra nil,\r\n   :complete\r\n   [{:name 36,\r\n     :start #<DateTime 2015-08-05T21:57:43.619Z>,\r\n     :end #<DateTime 2015-08-05T21:57:49.621Z>}\r\n    {:name 36,\r\n     :start #<DateTime 2015-08-05T21:58:06.665Z>,\r\n     :end #<DateTime 2015-08-05T21:58:12.666Z>}\r\n    {:name 36,\r\n     :start #<DateTime 2015-08-05T21:58:31.668Z>,\r\n     :end #<DateTime 2015-08-05T21:58:37.670Z>}],\r\n   :incomplete []},\r\n```\r\n\r\nThis job should have run 5 times, during a 6 second window recurring every 24 seconds, from 21:57:42 to 21:59:18. However, only the first 3 targets are satisfied--Chronos never invokes the final 2. You can download the full ZK, Mesos logs, operation history, and analysis [here](http://aphyr.com/media/chronos.tar.bz2).\r\n\r\nYou can reproduce these results by cloning [https://github.com/aphyr/jepsen/](jepsen), at commit c8a9acb274074efcb8bc689983f30ecac8f51605, setting up Jepsen (see https://github.com/aphyr/jepsen/tree/master/jepsen, or https://github.com/abailly/jepsen-vagrant), and running `lein test` in `jepsen/chronos`. I'm also happy to adjust the tests and re-run with newer versions.\r\n\r\nShould I adjust our expectations for the test? Is there additional debugging information that might be helpful? Anything else I can tell you about the test structure or environment? I'd like to get this figured out. :)"},"comment":{"url":"https://api.github.com/repos/mesos/chronos/issues/comments/128534563","html_url":"https://github.com/mesos/chronos/issues/508#issuecomment-128534563","issue_url":"https://api.github.com/repos/mesos/chronos/issues/508","id":128534563,"user":{"login":"brndnmtthws","id":3129093,"avatar_url":"https://avatars.githubusercontent.com/u/3129093?v=3","gravatar_id":"","url":"https://api.github.com/users/brndnmtthws","html_url":"https://github.com/brndnmtthws","followers_url":"https://api.github.com/users/brndnmtthws/followers","following_url":"https://api.github.com/users/brndnmtthws/following{/other_user}","gists_url":"https://api.github.com/users/brndnmtthws/gists{/gist_id}","starred_url":"https://api.github.com/users/brndnmtthws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brndnmtthws/subscriptions","organizations_url":"https://api.github.com/users/brndnmtthws/orgs","repos_url":"https://api.github.com/users/brndnmtthws/repos","events_url":"https://api.github.com/users/brndnmtthws/events{/privacy}","received_events_url":"https://api.github.com/users/brndnmtthws/received_events","type":"User","site_admin":false},"created_at":"2015-08-06T22:57:08Z","updated_at":"2015-08-06T22:57:08Z","body":"> Can I suggest that Chronos emit an error when asked to schedule operations at a finer granularity than the scheduler horizon, instead of accepting those jobs and failing to run them?\r\n\r\nGreat idea. Patches are welcome, too."}}