{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6648","labels_url":"https://api.github.com/repos/apache/spark/issues/6648/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6648/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6648/events","html_url":"https://github.com/apache/spark/pull/6648","id":85208680,"number":6648,"title":"[SPARK-8029][core][wip] shuffleoutput per attempt","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-06-04T18:01:49Z","updated_at":"2015-06-05T15:29:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6648","html_url":"https://github.com/apache/spark/pull/6648","diff_url":"https://github.com/apache/spark/pull/6648.diff","patch_url":"https://github.com/apache/spark/pull/6648.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-8029\r\n\r\nThis is a proof of concept for solving the issue by making each `ShuffleMapTask` attempt write to a different location.  `ShuffleBlockId` is extended to include the stage attempt id, so the fetch side knows which files to read from.  `MapStatus` also includes the stage attempt, so now there is one `MapStatus` per `(executor, attempt)` as opposed to one per `executor`.  This won't really matter when there is just one attempt per stage.  In a pathological case, you'd end up with one `MapStatus` per partition, which would be **much** worse, but that is very unlikely.\r\n\r\nI need to add in some more unit tests (I've put in some place holder TODOs), but I think this is ready for design / architecture level review.  It does pass my \"integration\" test.  (And when I include my changes to the DAGScheduler, it passes with the stricter criteria.)\r\n\r\ncc @JoshRosen "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/109329191","html_url":"https://github.com/apache/spark/pull/6648#issuecomment-109329191","issue_url":"https://api.github.com/repos/apache/spark/issues/6648","id":109329191,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-06-05T15:29:47Z","updated_at":"2015-06-05T15:29:47Z","body":"Build finished. Test FAILed."}}