{"action":"created","issue":{"url":"https://api.github.com/repos/mesos/chronos/issues/511","labels_url":"https://api.github.com/repos/mesos/chronos/issues/511/labels{/name}","comments_url":"https://api.github.com/repos/mesos/chronos/issues/511/comments","events_url":"https://api.github.com/repos/mesos/chronos/issues/511/events","html_url":"https://github.com/mesos/chronos/issues/511","id":99549542,"number":511,"title":"Jobs don't run when a network partition occurs","user":{"login":"aphyr","id":3748,"avatar_url":"https://avatars.githubusercontent.com/u/3748?v=3","gravatar_id":"","url":"https://api.github.com/users/aphyr","html_url":"https://github.com/aphyr","followers_url":"https://api.github.com/users/aphyr/followers","following_url":"https://api.github.com/users/aphyr/following{/other_user}","gists_url":"https://api.github.com/users/aphyr/gists{/gist_id}","starred_url":"https://api.github.com/users/aphyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aphyr/subscriptions","organizations_url":"https://api.github.com/users/aphyr/orgs","repos_url":"https://api.github.com/users/aphyr/repos","events_url":"https://api.github.com/users/aphyr/events{/privacy}","received_events_url":"https://api.github.com/users/aphyr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-08-07T00:13:38Z","updated_at":"2015-08-18T09:46:35Z","closed_at":null,"body":"Building on #508, in the presence of complete network partitions splitting the cluster in half, Chronos will give up executing, well, basically any tasks. For instance, [this run](https://aphyr.com/media/chronos3.tar.bz2) splits the network after 200 seconds, waits 200 seconds, heals, and so on, for 1000 seconds, followed by a 60 second stabilizing period and a final read. Essentially every job fails to meet its scheduling demands, whether enqueued before, during, or after failure. For example:\r\n\r\n```clj\r\n 10\r\n   {:valid? false,\r\n    :job\r\n    {:name 10,\r\n     :start #<DateTime 2015-08-06T23:45:31.016Z>,\r\n     :count 89,\r\n     :duration 5,\r\n     :epsilon 19,\r\n     :interval 36},\r\n    :solution nil,\r\n    :targets\r\n    ([#<DateTime 2015-08-06T23:45:31.016Z>\r\n      #<DateTime 2015-08-06T23:45:55.016Z>]\r\n     [#<DateTime 2015-08-06T23:46:07.016Z>\r\n      #<DateTime 2015-08-06T23:46:31.016Z>]\r\n     [#<DateTime 2015-08-06T23:46:43.016Z>\r\n      #<DateTime 2015-08-06T23:47:07.016Z>]\r\n     [#<DateTime 2015-08-06T23:47:19.016Z>\r\n      #<DateTime 2015-08-06T23:47:43.016Z>]\r\n     [#<DateTime 2015-08-06T23:47:55.016Z>\r\n      #<DateTime 2015-08-06T23:48:19.016Z>]\r\n     [#<DateTime 2015-08-06T23:48:31.016Z>\r\n      #<DateTime 2015-08-06T23:48:55.016Z>]\r\n     [#<DateTime 2015-08-06T23:49:07.016Z>\r\n      #<DateTime 2015-08-06T23:49:31.016Z>]\r\n     [#<DateTime 2015-08-06T23:49:43.016Z>\r\n      #<DateTime 2015-08-06T23:50:07.016Z>]\r\n     [#<DateTime 2015-08-06T23:50:19.016Z>\r\n      #<DateTime 2015-08-06T23:50:43.016Z>]\r\n     [#<DateTime 2015-08-06T23:50:55.016Z>\r\n      #<DateTime 2015-08-06T23:51:19.016Z>]\r\n     [#<DateTime 2015-08-06T23:51:31.016Z>\r\n      #<DateTime 2015-08-06T23:51:55.016Z>]\r\n     [#<DateTime 2015-08-06T23:52:07.016Z>\r\n      #<DateTime 2015-08-06T23:52:31.016Z>]\r\n     [#<DateTime 2015-08-06T23:52:43.016Z>\r\n      #<DateTime 2015-08-06T23:53:07.016Z>]\r\n     [#<DateTime 2015-08-06T23:53:19.016Z>\r\n      #<DateTime 2015-08-06T23:53:43.016Z>]\r\n     [#<DateTime 2015-08-06T23:53:55.016Z>\r\n      #<DateTime 2015-08-06T23:54:19.016Z>]\r\n     [#<DateTime 2015-08-06T23:54:31.016Z>\r\n      #<DateTime 2015-08-06T23:54:55.016Z>]\r\n     [#<DateTime 2015-08-06T23:55:07.016Z>\r\n      #<DateTime 2015-08-06T23:55:31.016Z>]\r\n     [#<DateTime 2015-08-06T23:55:43.016Z>\r\n      #<DateTime 2015-08-06T23:56:07.016Z>]\r\n     [#<DateTime 2015-08-06T23:56:19.016Z>\r\n      #<DateTime 2015-08-06T23:56:43.016Z>]\r\n     [#<DateTime 2015-08-06T23:56:55.016Z>\r\n      #<DateTime 2015-08-06T23:57:19.016Z>]\r\n     [#<DateTime 2015-08-06T23:57:31.016Z>\r\n      #<DateTime 2015-08-06T23:57:55.016Z>]\r\n     [#<DateTime 2015-08-06T23:58:07.016Z>\r\n      #<DateTime 2015-08-06T23:58:31.016Z>]\r\n     [#<DateTime 2015-08-06T23:58:43.016Z>\r\n      #<DateTime 2015-08-06T23:59:07.016Z>]\r\n     [#<DateTime 2015-08-06T23:59:19.016Z>\r\n      #<DateTime 2015-08-06T23:59:43.016Z>]\r\n     [#<DateTime 2015-08-06T23:59:55.016Z>\r\n      #<DateTime 2015-08-07T00:00:19.016Z>]\r\n     [#<DateTime 2015-08-07T00:00:31.016Z>\r\n      #<DateTime 2015-08-07T00:00:55.016Z>]\r\n     [#<DateTime 2015-08-07T00:01:07.016Z>\r\n      #<DateTime 2015-08-07T00:01:31.016Z>]\r\n     [#<DateTime 2015-08-07T00:01:43.016Z>\r\n      #<DateTime 2015-08-07T00:02:07.016Z>]),\r\n    :extra nil,\r\n    :complete\r\n    [{:node \"n5\",\r\n      :name 10,\r\n      :start #<DateTime 2015-08-06T23:45:31.686Z>,\r\n      :end #<DateTime 2015-08-06T23:45:36.689Z>}\r\n     {:node \"n4\",\r\n      :name 10,\r\n      :start #<DateTime 2015-08-06T23:46:08.794Z>,\r\n      :end #<DateTime 2015-08-06T23:46:13.797Z>}\r\n     {:node \"n4\",\r\n      :name 10,\r\n      :start #<DateTime 2015-08-06T23:46:44.792Z>,\r\n      :end #<DateTime 2015-08-06T23:46:49.794Z>}\r\n     {:node \"n5\",\r\n      :name 10,\r\n      :start #<DateTime 2015-08-06T23:47:20.827Z>,\r\n      :end #<DateTime 2015-08-06T23:47:25.828Z>}\r\n     {:node \"n5\",\r\n      :name 10,\r\n      :start #<DateTime 2015-08-06T23:47:55.834Z>,\r\n      :end #<DateTime 2015-08-06T23:48:00.836Z>}],\r\n    :incomplete []},\r\n```\r\n\r\nThis job simply gives up running after 23:47:55, when a network partition separates [n2 n3] from [n1 n4 n5]. Other tasks never run at all. Note that the cluster is happy to accept new jobs after this time, which suggests that some components of Chronos are still alive and recover from network failure after a few seconds of unavailability.\r\n\r\nIn the first partition, the Mesos slaves (n4, n5) are isolated from some (but not all) Mesos masters (n1, n2, n3)--and I'd understand if Mesos wasn't smart enough to elect a master with reachable slaves. However, when the network heals at ~23:51, I would expect Chronos to resume scheduling this job--and that doesn't seem to happen. Are there timeouts we should adjust?"},"comment":{"url":"https://api.github.com/repos/mesos/chronos/issues/comments/132146693","html_url":"https://github.com/mesos/chronos/issues/511#issuecomment-132146693","issue_url":"https://api.github.com/repos/mesos/chronos/issues/511","id":132146693,"user":{"login":"gkleiman","id":146374,"avatar_url":"https://avatars.githubusercontent.com/u/146374?v=3","gravatar_id":"","url":"https://api.github.com/users/gkleiman","html_url":"https://github.com/gkleiman","followers_url":"https://api.github.com/users/gkleiman/followers","following_url":"https://api.github.com/users/gkleiman/following{/other_user}","gists_url":"https://api.github.com/users/gkleiman/gists{/gist_id}","starred_url":"https://api.github.com/users/gkleiman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gkleiman/subscriptions","organizations_url":"https://api.github.com/users/gkleiman/orgs","repos_url":"https://api.github.com/users/gkleiman/repos","events_url":"https://api.github.com/users/gkleiman/events{/privacy}","received_events_url":"https://api.github.com/users/gkleiman/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T09:46:35Z","updated_at":"2015-08-18T09:46:35Z","body":"@aphyr: The initial Mesos Masters in [chronos8.tar.bz2](https://aphyr.com/media/chronos8.tar.bz2) die before the partitions resolve and there are no logs of the subsequent Mesos Master processes:\r\n\r\n* Mesos Master N1 dies 228 seconds into the test run.\r\n* Mesos Master N2 dies 353 seconds into the test run.\r\n* Mesos Master N3 dies 120 seconds into the test run, even before the first network partition happened. It was affected by the replicated log fetching issue.\r\n\r\nAre the logs of the other processes missing or did your restart logic fail?"}}