{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6648","labels_url":"https://api.github.com/repos/apache/spark/issues/6648/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6648/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6648/events","html_url":"https://github.com/apache/spark/pull/6648","id":85208680,"number":6648,"title":"[SPARK-8029][core] shuffleoutput per attempt","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":55,"created_at":"2015-06-04T18:01:49Z","updated_at":"2015-10-08T19:12:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6648","html_url":"https://github.com/apache/spark/pull/6648","diff_url":"https://github.com/apache/spark/pull/6648.diff","patch_url":"https://github.com/apache/spark/pull/6648.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-8029\r\n\r\nThis implements one of the approaches in the design doc on the jira: now each `ShuffleMapTask` attempt write to a different location.  `ShuffleBlockId` is extended to include the stage attempt id, so the fetch side knows which files to read from.  `MapStatus` also includes the stage attempt, so now there is one `MapStatus` per `(executor, attempt)` as opposed to one per `executor`.  This won't really matter when there is just one attempt per stage.  In a pathological case, you'd end up with one `MapStatus` per partition, which would be **much** worse, but that is very unlikely.\r\n\r\nThis touches a lot of files, but almost all of the changes are just plumbing a `stageAttemptId` through a *lot* of different places.\r\n\r\ncc @JoshRosen "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/146659319","html_url":"https://github.com/apache/spark/pull/6648#issuecomment-146659319","issue_url":"https://api.github.com/repos/apache/spark/issues/6648","id":146659319,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-10-08T19:12:29Z","updated_at":"2015-10-08T19:12:29Z","body":"  [Test build #43401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43401/console) for   PR 6648 at commit [`f37be91`](https://github.com/apache/spark/commit/f37be9120499d699a2da34cb69620cd8dba9cb8a).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)`\n  * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n  * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n"}}