{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8887","labels_url":"https://api.github.com/repos/apache/spark/issues/8887/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8887/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8887/events","html_url":"https://github.com/apache/spark/pull/8887","id":108009077,"number":8887,"title":"[SPARK-10622] [core] Differentiate dead from \"mostly dead\" executors.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-23T21:21:18Z","updated_at":"2015-09-23T21:52:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8887","html_url":"https://github.com/apache/spark/pull/8887","diff_url":"https://github.com/apache/spark/pull/8887.diff","patch_url":"https://github.com/apache/spark/pull/8887.patch"},"body":"In YARN mode, when preemption is enabled, we may leave executors in a\r\nzombie state while we wait to retrieve the reason for which the executor\r\nexited. This is so that we don't account for failed tasks that were\r\nrunning on a preempted executor.\r\n\r\nThe issue is that while we wait for this information, the scheduler\r\nmight decide to schedule tasks on the executor, which will never be\r\nable to run them. Other side effects include the block manager still\r\nconsidering the executor available to cache blocks, for example.\r\n\r\nSo, when we know that an executor went down but we don't know why,\r\nstop everything related to the executor, except its running tasks.\r\nOnly when we know the reason for the exit (or give up waiting for\r\nit) we do update the running tasks.\r\n\r\nThis is achieved by a new `disableExecutor()` method in the\r\n`Schedulable` interface. For managers that do not behave like this\r\n(i.e. every one but YARN), the existing `executorLost()` method\r\nwill behave the same way it did before.\r\n\r\nOn top of that change, a few minor changes that made debugging easier,\r\nand fixed some other minor issues:\r\n- The cluster-mode AM was printing a misleading log message every\r\n  time an executor disconnected from the driver (because the akka\r\n  actor system was shared between driver and AM).\r\n- Avoid sending unnecessary requests for an executor's exit reason\r\n  when we already know it was explicitly disabled / killed. This\r\n  avoids both multiple requests, and unnecessary requests that would\r\n  just cause warning messages on the AM (in the explicit kill case).\r\n- Tone down a log message about the executor being lost when it\r\n  exited normally (e.g. preemption)\r\n- Wake up the AM monitor thread when requests for executor loss\r\n  reasons arrive too, so that we can more quickly remove executors\r\n  from this zombie state."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/142740970","html_url":"https://github.com/apache/spark/pull/8887#issuecomment-142740970","issue_url":"https://api.github.com/repos/apache/spark/issues/8887","id":142740970,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T21:52:12Z","updated_at":"2015-09-23T21:52:12Z","body":" Merged build triggered."}}