{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8280","labels_url":"https://api.github.com/repos/apache/spark/issues/8280/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8280/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8280/events","html_url":"https://github.com/apache/spark/pull/8280","id":101718088,"number":8280,"title":"[SPARK-10087] [CORE] Disable spark.shuffle.reduceLocality.enabled by default.","user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-08-18T18:17:00Z","updated_at":"2015-08-18T19:52:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8280","html_url":"https://github.com/apache/spark/pull/8280","diff_url":"https://github.com/apache/spark/pull/8280.diff","patch_url":"https://github.com/apache/spark/pull/8280.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-10087\r\n\r\nIn some cases, when spark.shuffle.reduceLocality.enabled is enabled, we are scheduling all reducers to the same executor (the cluster has plenty of resources). Changing spark.shuffle.reduceLocality.enabled to false resolve the problem. \r\n\r\nHere is a little bit more information. For one of my query, all 200 reducers were scheduled to the same reducer and every reducer has about 800 KB input.\r\n "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/132331662","html_url":"https://github.com/apache/spark/pull/8280#issuecomment-132331662","issue_url":"https://api.github.com/repos/apache/spark/issues/8280","id":132331662,"user":{"login":"shivaram","id":143893,"avatar_url":"https://avatars.githubusercontent.com/u/143893?v=3","gravatar_id":"","url":"https://api.github.com/users/shivaram","html_url":"https://github.com/shivaram","followers_url":"https://api.github.com/users/shivaram/followers","following_url":"https://api.github.com/users/shivaram/following{/other_user}","gists_url":"https://api.github.com/users/shivaram/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaram/subscriptions","organizations_url":"https://api.github.com/users/shivaram/orgs","repos_url":"https://api.github.com/users/shivaram/repos","events_url":"https://api.github.com/users/shivaram/events{/privacy}","received_events_url":"https://api.github.com/users/shivaram/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T19:52:49Z","updated_at":"2015-08-18T19:52:49Z","body":"So my hypothesis right now is that the RDD in the reduce stage has two Shuffle dependencies and the first shuffle dependency happens to be the single map task stage -- so the locality preference ends up giving all the tasks to the single host.\r\n\r\nHmm so my guess is that we need to be able to differentiate among different shuffle dependencies ideally. Here is another suggestion: Can we turn this off if we have more than one shuffle dependency ? it should be pretty cheap to count that "}}