{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8760","labels_url":"https://api.github.com/repos/apache/spark/issues/8760/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8760/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8760/events","html_url":"https://github.com/apache/spark/pull/8760","id":106491278,"number":8760,"title":"[Spark-8426] [scheduler] enhance blacklist mechanism","user":{"login":"mwws","id":13216322,"avatar_url":"https://avatars.githubusercontent.com/u/13216322?v=3","gravatar_id":"","url":"https://api.github.com/users/mwws","html_url":"https://github.com/mwws","followers_url":"https://api.github.com/users/mwws/followers","following_url":"https://api.github.com/users/mwws/following{/other_user}","gists_url":"https://api.github.com/users/mwws/gists{/gist_id}","starred_url":"https://api.github.com/users/mwws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwws/subscriptions","organizations_url":"https://api.github.com/users/mwws/orgs","repos_url":"https://api.github.com/users/mwws/repos","events_url":"https://api.github.com/users/mwws/events{/privacy}","received_events_url":"https://api.github.com/users/mwws/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-09-15T06:40:58Z","updated_at":"2015-10-19T01:54:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8760","html_url":"https://github.com/apache/spark/pull/8760","diff_url":"https://github.com/apache/spark/pull/8760.diff","patch_url":"https://github.com/apache/spark/pull/8760.patch"},"body":"1. add a global BlacklistTracker which is able to share blacklisted executors information across TaskSetManager. Different BlacklistStrategy could be applied to determine which executors should be put in blacklist. And it's convenient to add new strategy later according to current design.\r\n2. remove duplicate \"failedExecutors\" list in TaskSetManager and make BlacklistTracker being the only interface for blacklist mechanism in codebase which is logically cleaner.\r\n3. for backward compatibility, implement DefaultStrategy to keep the same semantics as current by default \r\n4. apply BlacklistTracker in YARN to avoid allocating new executors on blacklisted node, the blacklisted node information will be passed to YARN ResourceManager. it affects in dynamic allocation mode.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/149076636","html_url":"https://github.com/apache/spark/pull/8760#issuecomment-149076636","issue_url":"https://api.github.com/repos/apache/spark/issues/8760","id":149076636,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-10-19T01:54:22Z","updated_at":"2015-10-19T01:54:22Z","body":"  [Test build #43908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43908/consoleFull) for   PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5)."}}