{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/40298903","id":40298903,"diff_hunk":"@@ -0,0 +1,156 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one or more\n+ * contributor license agreements.  See the NOTICE file distributed with\n+ * this work for additional information regarding copyright ownership.\n+ * The ASF licenses this file to You under the Apache License, Version 2.0\n+ * (the \"License\"); you may not use this file except in compliance with\n+ * the License.  You may obtain a copy of the License at\n+ *\n+ *    http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+\n+package org.apache.spark.scheduler\n+\n+import scala.collection.mutable\n+\n+import org.apache.spark.Logging\n+import org.apache.spark.SparkConf\n+import org.apache.spark.util.SystemClock\n+\n+/**\n+ * The interface to determine executor blacklist and node blacklist.\n+ */\n+trait BlacklistStrategy {\n+  val expireTimeInMillisecond: Long\n+\n+  // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId","path":"core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala","position":null,"original_position":32,"commit_id":"3e04fe7ec695df6849d910158ebf1d9a8566e4fe","original_commit_id":"24cdf30d6144d084b5c8ff23984e037bab18b4dd","user":{"login":"mwws","id":13216322,"avatar_url":"https://avatars.githubusercontent.com/u/13216322?v=3","gravatar_id":"","url":"https://api.github.com/users/mwws","html_url":"https://github.com/mwws","followers_url":"https://api.github.com/users/mwws/followers","following_url":"https://api.github.com/users/mwws/following{/other_user}","gists_url":"https://api.github.com/users/mwws/gists{/gist_id}","starred_url":"https://api.github.com/users/mwws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwws/subscriptions","organizations_url":"https://api.github.com/users/mwws/orgs","repos_url":"https://api.github.com/users/mwws/repos","events_url":"https://api.github.com/users/mwws/events{/privacy}","received_events_url":"https://api.github.com/users/mwws/received_events","type":"User","site_admin":false},"body":"Rethink about it, for \"speculate\" mode, we can also call taskId unrelated ```getNodeBlacklist```, it should be fine. And I have simplified the API as you suggested.","created_at":"2015-09-24T09:11:01Z","updated_at":"2015-09-24T09:11:01Z","html_url":"https://github.com/apache/spark/pull/8760#discussion_r40298903","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/8760","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/40298903"},"html":{"href":"https://github.com/apache/spark/pull/8760#discussion_r40298903"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/8760"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8760","id":45013815,"html_url":"https://github.com/apache/spark/pull/8760","diff_url":"https://github.com/apache/spark/pull/8760.diff","patch_url":"https://github.com/apache/spark/pull/8760.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/8760","number":8760,"state":"open","locked":false,"title":"[Spark-8426] [scheduler] enhance blacklist mechanism","user":{"login":"mwws","id":13216322,"avatar_url":"https://avatars.githubusercontent.com/u/13216322?v=3","gravatar_id":"","url":"https://api.github.com/users/mwws","html_url":"https://github.com/mwws","followers_url":"https://api.github.com/users/mwws/followers","following_url":"https://api.github.com/users/mwws/following{/other_user}","gists_url":"https://api.github.com/users/mwws/gists{/gist_id}","starred_url":"https://api.github.com/users/mwws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwws/subscriptions","organizations_url":"https://api.github.com/users/mwws/orgs","repos_url":"https://api.github.com/users/mwws/repos","events_url":"https://api.github.com/users/mwws/events{/privacy}","received_events_url":"https://api.github.com/users/mwws/received_events","type":"User","site_admin":false},"body":"1. add a global BlacklistTracker which is able to share blacklisted executors information across TaskSetManager. Different BlacklistStrategy could be applied to determine which executors should be put in blacklist. And it's convenient to add new strategy later according to current design.\r\n2. remove duplicate \"failedExecutors\" list in TaskSetManager and make BlacklistTracker being the only interface for blacklist mechanism in codebase which is logically cleaner.\r\n3. for backward compatibility, implement DefaultStrategy to keep the same semantics as current by default \r\n4. apply BlacklistTracker in YARN to avoid allocating new executors on blacklisted node, the blacklisted node information will be passed to YARN ResourceManager. it affects in dynamic allocation mode.\r\n","created_at":"2015-09-15T06:40:58Z","updated_at":"2015-09-24T09:11:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"03ce147e63913a064b50739d56df9a7682b23819","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/8760/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/8760/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/8760/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/3e04fe7ec695df6849d910158ebf1d9a8566e4fe","head":{"label":"mwws:SPARK-8426","ref":"SPARK-8426","sha":"3e04fe7ec695df6849d910158ebf1d9a8566e4fe","user":{"login":"mwws","id":13216322,"avatar_url":"https://avatars.githubusercontent.com/u/13216322?v=3","gravatar_id":"","url":"https://api.github.com/users/mwws","html_url":"https://github.com/mwws","followers_url":"https://api.github.com/users/mwws/followers","following_url":"https://api.github.com/users/mwws/following{/other_user}","gists_url":"https://api.github.com/users/mwws/gists{/gist_id}","starred_url":"https://api.github.com/users/mwws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwws/subscriptions","organizations_url":"https://api.github.com/users/mwws/orgs","repos_url":"https://api.github.com/users/mwws/repos","events_url":"https://api.github.com/users/mwws/events{/privacy}","received_events_url":"https://api.github.com/users/mwws/received_events","type":"User","site_admin":false},"repo":{"id":38867540,"name":"spark","full_name":"mwws/spark","owner":{"login":"mwws","id":13216322,"avatar_url":"https://avatars.githubusercontent.com/u/13216322?v=3","gravatar_id":"","url":"https://api.github.com/users/mwws","html_url":"https://github.com/mwws","followers_url":"https://api.github.com/users/mwws/followers","following_url":"https://api.github.com/users/mwws/following{/other_user}","gists_url":"https://api.github.com/users/mwws/gists{/gist_id}","starred_url":"https://api.github.com/users/mwws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwws/subscriptions","organizations_url":"https://api.github.com/users/mwws/orgs","repos_url":"https://api.github.com/users/mwws/repos","events_url":"https://api.github.com/users/mwws/events{/privacy}","received_events_url":"https://api.github.com/users/mwws/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mwws/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/mwws/spark","forks_url":"https://api.github.com/repos/mwws/spark/forks","keys_url":"https://api.github.com/repos/mwws/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mwws/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mwws/spark/teams","hooks_url":"https://api.github.com/repos/mwws/spark/hooks","issue_events_url":"https://api.github.com/repos/mwws/spark/issues/events{/number}","events_url":"https://api.github.com/repos/mwws/spark/events","assignees_url":"https://api.github.com/repos/mwws/spark/assignees{/user}","branches_url":"https://api.github.com/repos/mwws/spark/branches{/branch}","tags_url":"https://api.github.com/repos/mwws/spark/tags","blobs_url":"https://api.github.com/repos/mwws/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mwws/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mwws/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/mwws/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mwws/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/mwws/spark/languages","stargazers_url":"https://api.github.com/repos/mwws/spark/stargazers","contributors_url":"https://api.github.com/repos/mwws/spark/contributors","subscribers_url":"https://api.github.com/repos/mwws/spark/subscribers","subscription_url":"https://api.github.com/repos/mwws/spark/subscription","commits_url":"https://api.github.com/repos/mwws/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/mwws/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/mwws/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/mwws/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/mwws/spark/contents/{+path}","compare_url":"https://api.github.com/repos/mwws/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mwws/spark/merges","archive_url":"https://api.github.com/repos/mwws/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mwws/spark/downloads","issues_url":"https://api.github.com/repos/mwws/spark/issues{/number}","pulls_url":"https://api.github.com/repos/mwws/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/mwws/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/mwws/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mwws/spark/labels{/name}","releases_url":"https://api.github.com/repos/mwws/spark/releases{/id}","created_at":"2015-07-10T07:35:43Z","updated_at":"2015-09-02T07:35:19Z","pushed_at":"2015-09-24T09:04:04Z","git_url":"git://github.com/mwws/spark.git","ssh_url":"git@github.com:mwws/spark.git","clone_url":"https://github.com/mwws/spark.git","svn_url":"https://github.com/mwws/spark","homepage":null,"size":124815,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"833be73314b85b390a9007ed6ed63dc47bbd9e4f","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-09-24T08:43:22Z","pushed_at":"2015-09-24T09:09:36Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1626186,"stargazers_count":5528,"watchers_count":5528,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5107,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":335,"forks":5107,"open_issues":335,"watchers":5528,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/8760"},"html":{"href":"https://github.com/apache/spark/pull/8760"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/8760"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/8760/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/8760/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/8760/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/3e04fe7ec695df6849d910158ebf1d9a8566e4fe"}}}}