{"action":"created","issue":{"url":"https://api.github.com/repos/mesos/chronos/issues/513","labels_url":"https://api.github.com/repos/mesos/chronos/issues/513/labels{/name}","comments_url":"https://api.github.com/repos/mesos/chronos/issues/513/comments","events_url":"https://api.github.com/repos/mesos/chronos/issues/513/events","html_url":"https://github.com/mesos/chronos/issues/513","id":99709536,"number":513,"title":"A partition isolating Chronos from the ZK leader can cause a crash","user":{"login":"aphyr","id":3748,"avatar_url":"https://avatars.githubusercontent.com/u/3748?v=3","gravatar_id":"","url":"https://api.github.com/users/aphyr","html_url":"https://github.com/aphyr","followers_url":"https://api.github.com/users/aphyr/followers","following_url":"https://api.github.com/users/aphyr/following{/other_user}","gists_url":"https://api.github.com/users/aphyr/gists{/gist_id}","starred_url":"https://api.github.com/users/aphyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aphyr/subscriptions","organizations_url":"https://api.github.com/users/aphyr/orgs","repos_url":"https://api.github.com/users/aphyr/repos","events_url":"https://api.github.com/users/aphyr/events{/privacy}","received_events_url":"https://api.github.com/users/aphyr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-08-07T19:07:51Z","updated_at":"2015-08-11T00:33:40Z","closed_at":"2015-08-11T00:33:40Z","body":"When a network partition isolates a Chronos node from the Zookeeper leader, the Chronos process may exit entirely, resulting in downtime until an operator intervenes to restart it.\r\n\r\n```\r\nAug  7 11:55:26 n5 chronos[48789]: [2015-08-07 11:55:26,752] INFO Client session timed out, have not heard from server in 26666ms for sessionid 0x24f09828c5e0001, closing socket connection \r\nand attempting reconnect (org.apache.zookeeper.ClientCnxn:1096)\r\nAug  7 11:55:26 n5 chronos[48789]: [2015-08-07 11:55:26,853] INFO State change: SUSPENDED (org.apache.curator.framework.state.ConnectionStateManager:228)\r\nAug  7 11:55:26 n5 chronos[48789]: I0807 11:55:26.854676 48930 sched.cpp:1591] Asked to stop the driver\r\nAug  7 11:55:26 n5 chronos[48789]: [2015-08-07 11:55:26,854] INFO Defeated. Not the current leader. (org.apache.mesos.chronos.scheduler.jobs.JobScheduler:583)\r\nAug  7 11:55:27 n5 chronos[48789]: [2015-08-07 11:55:27,354] INFO Opening socket connection to server n2/192.168.122.12:2181. Will not attempt to authenticate using SASL (unknown error) (or\r\ng.apache.zookeeper.ClientCnxn:975)\r\nAug  7 11:55:34 n5 chronos[48789]: [2015-08-07 11:55:34,860] INFO Client session timed out, have not heard from server in 8007ms for sessionid 0x24f09828c5e0001, closing socket connection a\r\nnd attempting reconnect (org.apache.zookeeper.ClientCnxn:1096)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,164] INFO Opening socket connection to server n1/192.168.122.11:2181. Will not attempt to authenticate using SASL (unknown error) (or\r\ng.apache.zookeeper.ClientCnxn:975)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,165] INFO Socket connection established to n1/192.168.122.11:2181, initiating session (org.apache.zookeeper.ClientCnxn:852)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,168] INFO Session establishment complete on server n1/192.168.122.11:2181, sessionid = 0x24f09828c5e0001, negotiated timeout = 40000 \r\n(org.apache.zookeeper.ClientCnxn:1235)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,168] INFO State change: RECONNECTED (org.apache.curator.framework.state.ConnectionStateManager:228)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,180] ERROR Error trying to talk to zookeeper. Exiting. (org.apache.mesos.chronos.scheduler.jobs.JobScheduler:66)\r\nAug  7 11:55:35 n5 chronos[48789]: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /chronos/state/candidate/_c_ece6cabb-24de-447d-91a2-0ffb7749f750-latch-\r\n0000000000\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:374)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:345)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:449)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.mesos.chronos.scheduler.jobs.JobScheduler.getLeader(JobScheduler.scala:63)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.mesos.chronos.scheduler.api.RedirectFilter.doFilter(RedirectFilter.scala:29)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.codahale.metrics.jetty8.InstrumentedHandler.handle(InstrumentedHandler.java:192)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.Server.handle(Server.java:370)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at java.lang.Thread.run(Thread.java:745)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,183] INFO Task 'ct:1438973701557:0:30:' launched, status: 'DRIVER_RUNNING' (org.apache.mesos.chronos.scheduler.mesos.MesosJobFramework:184)\r\nAug  7 11:55:35 n5 chronos[48789]: E0807 11:55:35.185912 48907 socket.hpp:107] Shutdown failed on fd=72: Transport endpoint is not connected [107]\r\nAug  7 11:55:35 n5 chronos[48789]: I0807 11:55:35.185976 48888 sched.cpp:835] Stopping framework '20150807-115313-226142400-5050-33792-0000'\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,182] ERROR Error trying to talk to zookeeper. Exiting. (org.apache.mesos.chronos.scheduler.jobs.JobScheduler:66)\r\nAug  7 11:55:35 n5 chronos[48789]: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /chronos/state/candidate/_c_ece6cabb-24de-447d-91a2-0ffb7749f750-latch-0000000000\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:374)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:345)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:449)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.mesos.chronos.scheduler.jobs.JobScheduler.getLeader(JobScheduler.scala:63)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.apache.mesos.chronos.scheduler.api.RedirectFilter.doFilter(RedirectFilter.scala:29)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at com.codahale.metrics.jetty8.InstrumentedHandler.handle(InstrumentedHandler.java:192)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.Server.handle(Server.java:370)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\r\nAug  7 11:55:35 n5 chronos[48789]: #011at java.lang.Thread.run(Thread.java:745)\r\n\r\n... more \"Error trying to talk to zookeeper. Exiting\" stacktraces ...\r\n\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,185] INFO Shutting down services (org.apache.mesos.chronos.scheduler.Main$:42)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,192] INFO Waiting for services to shut down (org.apache.mesos.chronos.scheduler.Main$:48)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,199] INFO Session: 0x24f09828c5e0001 closed (org.apache.zookeeper.ZooKeeper:684)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,199] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn:512)\r\nAug  7 11:55:35 n5 chronos[48789]: [2015-08-07 11:55:35,286] INFO stopped o.e.j.s.ServletContextHandler{/,null} (org.eclipse.jetty.server.handler.ContextHandler:843)\r\n```"},"comment":{"url":"https://api.github.com/repos/mesos/chronos/issues/comments/129657139","html_url":"https://github.com/mesos/chronos/issues/513#issuecomment-129657139","issue_url":"https://api.github.com/repos/mesos/chronos/issues/513","id":129657139,"user":{"login":"brndnmtthws","id":3129093,"avatar_url":"https://avatars.githubusercontent.com/u/3129093?v=3","gravatar_id":"","url":"https://api.github.com/users/brndnmtthws","html_url":"https://github.com/brndnmtthws","followers_url":"https://api.github.com/users/brndnmtthws/followers","following_url":"https://api.github.com/users/brndnmtthws/following{/other_user}","gists_url":"https://api.github.com/users/brndnmtthws/gists{/gist_id}","starred_url":"https://api.github.com/users/brndnmtthws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brndnmtthws/subscriptions","organizations_url":"https://api.github.com/users/brndnmtthws/orgs","repos_url":"https://api.github.com/users/brndnmtthws/repos","events_url":"https://api.github.com/users/brndnmtthws/events{/privacy}","received_events_url":"https://api.github.com/users/brndnmtthws/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T00:33:40Z","updated_at":"2015-08-11T00:33:40Z","body":"This is not the forum for that discussion. From what I can tell, the code functioned as intended. If Chronos can't talk to ZK, it will terminate."}}