{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8923","labels_url":"https://api.github.com/repos/apache/spark/issues/8923/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8923/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8923/events","html_url":"https://github.com/apache/spark/pull/8923","id":108463286,"number":8923,"title":"[SPARK][SPARK-10842]Eliminate creating duplicate stage while generate job dag","user":{"login":"suyanNone","id":9818852,"avatar_url":"https://avatars.githubusercontent.com/u/9818852?v=3","gravatar_id":"","url":"https://api.github.com/users/suyanNone","html_url":"https://github.com/suyanNone","followers_url":"https://api.github.com/users/suyanNone/followers","following_url":"https://api.github.com/users/suyanNone/following{/other_user}","gists_url":"https://api.github.com/users/suyanNone/gists{/gist_id}","starred_url":"https://api.github.com/users/suyanNone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyanNone/subscriptions","organizations_url":"https://api.github.com/users/suyanNone/orgs","repos_url":"https://api.github.com/users/suyanNone/repos","events_url":"https://api.github.com/users/suyanNone/events{/privacy}","received_events_url":"https://api.github.com/users/suyanNone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-09-26T13:15:20Z","updated_at":"2015-11-13T16:14:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8923","html_url":"https://github.com/apache/spark/pull/8923","diff_url":"https://github.com/apache/spark/pull/8923.diff","patch_url":"https://github.com/apache/spark/pull/8923.patch"},"body":"When we traverse RDD, to generate Stage DAG, Spark will skip to judge the stage was added into shuffleIdToStage or not in some condition: get shuffleDep from getAncestorShuffleDependency\r\n\r\n\r\nBefore:\r\n![before1](https://cloud.githubusercontent.com/assets/9818852/10117739/9e1e8778-6493-11e5-935d-ea32099e94c9.png)\r\n![before2](https://cloud.githubusercontent.com/assets/9818852/10117740/9e20ee46-6493-11e5-95b4-f30df5ac22e1.png)\r\n\r\n\r\n\r\nAfter:\r\n![after](https://cloud.githubusercontent.com/assets/9818852/10117737/9e1864c4-6493-11e5-8541-31ff0ead329a.png)\r\n![after2](https://cloud.githubusercontent.com/assets/9818852/10117738/9e1c8658-6493-11e5-954a-fb00f36f4d3b.png)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/156476494","html_url":"https://github.com/apache/spark/pull/8923#issuecomment-156476494","issue_url":"https://api.github.com/repos/apache/spark/issues/8923","id":156476494,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-11-13T16:14:47Z","updated_at":"2015-11-13T16:14:47Z","body":"  [Test build #45867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45867/consoleFull) for   PR 8923 at commit [`a1aa8be`](https://github.com/apache/spark/commit/a1aa8be56ae42c5f6b1d1afe0a82c2c4b41e9fcd)."}}