{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4708","labels_url":"https://api.github.com/repos/apache/spark/issues/4708/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4708/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4708/events","html_url":"https://github.com/apache/spark/pull/4708","id":58416613,"number":4708,"title":"[SPARK-4655] Split Stage into ShuffleMapStage and ResultStage subclasses","user":{"login":"ilganeli","id":1483345,"avatar_url":"https://avatars.githubusercontent.com/u/1483345?v=3","gravatar_id":"","url":"https://api.github.com/users/ilganeli","html_url":"https://github.com/ilganeli","followers_url":"https://api.github.com/users/ilganeli/followers","following_url":"https://api.github.com/users/ilganeli/following{/other_user}","gists_url":"https://api.github.com/users/ilganeli/gists{/gist_id}","starred_url":"https://api.github.com/users/ilganeli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilganeli/subscriptions","organizations_url":"https://api.github.com/users/ilganeli/orgs","repos_url":"https://api.github.com/users/ilganeli/repos","events_url":"https://api.github.com/users/ilganeli/events{/privacy}","received_events_url":"https://api.github.com/users/ilganeli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":43,"created_at":"2015-02-20T22:01:35Z","updated_at":"2015-02-24T22:54:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4708","html_url":"https://github.com/apache/spark/pull/4708","diff_url":"https://github.com/apache/spark/pull/4708.diff","patch_url":"https://github.com/apache/spark/pull/4708.patch"},"body":"Hi all - this patch includes splitting up Stage into ShuffleMapStage and ResultStage and updating their usage within DAGScheduler.\r\n\r\nI wanted to confirm that it's appropriate to move the outputLocs variable and its associated methods into ShuffleMapStage. Within the shuffleMapTask handler (line 1004) I wanted to confirm that we are guaranteed for that stage to be a ShuffleMapStage\r\n\r\nI believe I've split up the functionality of ResultStage and ShuffleMapStage appropriately but I wanted to make sure I'm not missing something.\r\n\r\nLastly, I suggest opening a JIRA to further improve readability of two functions within the DAGScheduler:\r\n\r\n1) The monolithic handleTaskCompletion \r\n2) The cleanupStateForJobAndIndependentStages function\r\n\r\nThis last item is already code-complete in a separate PR but it should really be done within its own issue. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/75865176","html_url":"https://github.com/apache/spark/pull/4708#issuecomment-75865176","issue_url":"https://api.github.com/repos/apache/spark/issues/4708","id":75865176,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-02-24T22:54:13Z","updated_at":"2015-02-24T22:54:13Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27912/\nTest FAILed."}}