{"action":"opened","issue":{"url":"https://api.github.com/repos/mesos/mesos-go/issues/155","labels_url":"https://api.github.com/repos/mesos/mesos-go/issues/155/labels{/name}","comments_url":"https://api.github.com/repos/mesos/mesos-go/issues/155/comments","events_url":"https://api.github.com/repos/mesos/mesos-go/issues/155/events","html_url":"https://github.com/mesos/mesos-go/issues/155","id":100419172,"number":155,"title":"Http transports blocks for messages sent after a master died but before failover","user":{"login":"sttts","id":730123,"avatar_url":"https://avatars.githubusercontent.com/u/730123?v=3","gravatar_id":"","url":"https://api.github.com/users/sttts","html_url":"https://github.com/sttts","followers_url":"https://api.github.com/users/sttts/followers","following_url":"https://api.github.com/users/sttts/following{/other_user}","gists_url":"https://api.github.com/users/sttts/gists{/gist_id}","starred_url":"https://api.github.com/users/sttts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sttts/subscriptions","organizations_url":"https://api.github.com/users/sttts/orgs","repos_url":"https://api.github.com/users/sttts/repos","events_url":"https://api.github.com/users/sttts/events{/privacy}","received_events_url":"https://api.github.com/users/sttts/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-08-11T21:46:39Z","updated_at":"2015-08-11T21:46:39Z","closed_at":null,"body":"**Context:** Kubernetes on Mesos scheduler\r\n\r\nThe scheduler does not survive Mesos master failover most of the times. \r\n\r\nA log is shown in this gist: https://gist.github.com/sttts/fd66462513f6630aeeb7. Also compare https://github.com/GoogleCloudPlatform/kubernetes/issues/12281.\r\n\r\n**Analysis:** The scheduler's http client is blocked by a message sent to the master after the master failed, but before the scheduler knew about that. In the case of the log above it takes several seconds until a \"No route to host\" error is returned. During this time, the scheduler cannot reregister with the master, i.e. the `ReregisterFrameworkMessage` messages are actually never sent.\r\n\r\nEventually the timeout happens and the \"No route to host\" error is shown. Unfortunately the Kubernetes scheduler terminates due to this. The example_scheduler recovers.\r\n\r\n**Test case:** The same blocking http sending queue can be triggered by adding a `time.Sleep(3*time.Second)` to the `ResourceOffers` function of `example_scheduler.go`. Start this scheduler against e.g. a 3 master Mesos setup with Zookeeper URLs and shutdown the active master. If the `LaunchTasks` message is sent before the scheduler notices the dead master, i.e. before `zk.EventNodeChildrenChanged` is shown, the following reregistrations will fail."}}