{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4703","labels_url":"https://api.github.com/repos/apache/spark/issues/4703/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4703/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4703/events","html_url":"https://github.com/apache/spark/pull/4703","id":58319814,"number":4703,"title":"[SPARK-4655] Split Stage into ShuffleMapStage and ResultStage subclasses","user":{"login":"ilganeli","id":1483345,"avatar_url":"https://avatars.githubusercontent.com/u/1483345?v=3","gravatar_id":"","url":"https://api.github.com/users/ilganeli","html_url":"https://github.com/ilganeli","followers_url":"https://api.github.com/users/ilganeli/followers","following_url":"https://api.github.com/users/ilganeli/following{/other_user}","gists_url":"https://api.github.com/users/ilganeli/gists{/gist_id}","starred_url":"https://api.github.com/users/ilganeli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilganeli/subscriptions","organizations_url":"https://api.github.com/users/ilganeli/orgs","repos_url":"https://api.github.com/users/ilganeli/repos","events_url":"https://api.github.com/users/ilganeli/events{/privacy}","received_events_url":"https://api.github.com/users/ilganeli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-02-20T07:47:41Z","updated_at":"2015-02-20T18:02:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4703","html_url":"https://github.com/apache/spark/pull/4703","diff_url":"https://github.com/apache/spark/pull/4703.diff","patch_url":"https://github.com/apache/spark/pull/4703.patch"},"body":"Hi all - this patch includes two main efforts:\r\n\r\n1) I've split up Stage into ShuffleMapStage and ResultStage and updated their usage within DAGScheduler\r\n2) While doing this, I ended up cleaning up a lot of the long and awkward functions within the DAGScheduler since I needed to do that to make sense of the code. I believe this improved readability of the code significantly - the only problem is that it's not straightforward to diff those changes side by side (particularly the updates to handleFetchFailure(). \r\n\r\nI wanted to confirm that it's appropriate to move the outputLocs variable and its associated methods into ShuffleMapStage. The one function of concern is within the completeStage() and handleFailedTasks() functions where I wanted to confirm that we are guaranteed for that stage to be a ShuffleMapStage\r\n\r\nI believe I've split up the functionality of ResultStage and ShuffleMapStage appropriately but I wanted to make sure I'm not missing something."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/75288542","html_url":"https://github.com/apache/spark/pull/4703#issuecomment-75288542","issue_url":"https://api.github.com/repos/apache/spark/issues/4703","id":75288542,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-02-20T18:02:33Z","updated_at":"2015-02-20T18:02:33Z","body":"  [Test build #27783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27783/consoleFull) for   PR 4703 at commit [`0a4109e`](https://github.com/apache/spark/commit/0a4109ecb1982130e7f561e8f72d47c6fd0cd9e0).\n * This patch **does not merge cleanly**."}}