{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8180","labels_url":"https://api.github.com/repos/apache/spark/issues/8180/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8180/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8180/events","html_url":"https://github.com/apache/spark/pull/8180","id":100880342,"number":8180,"title":"[SPARK-9851] Support submitting map stages individually in DAGScheduler","user":{"login":"mateiz","id":228859,"avatar_url":"https://avatars.githubusercontent.com/u/228859?v=3","gravatar_id":"","url":"https://api.github.com/users/mateiz","html_url":"https://github.com/mateiz","followers_url":"https://api.github.com/users/mateiz/followers","following_url":"https://api.github.com/users/mateiz/following{/other_user}","gists_url":"https://api.github.com/users/mateiz/gists{/gist_id}","starred_url":"https://api.github.com/users/mateiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mateiz/subscriptions","organizations_url":"https://api.github.com/users/mateiz/orgs","repos_url":"https://api.github.com/users/mateiz/repos","events_url":"https://api.github.com/users/mateiz/events{/privacy}","received_events_url":"https://api.github.com/users/mateiz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-08-13T22:04:39Z","updated_at":"2015-08-13T22:51:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8180","html_url":"https://github.com/apache/spark/pull/8180","diff_url":"https://github.com/apache/spark/pull/8180.diff","patch_url":"https://github.com/apache/spark/pull/8180.patch"},"body":"This patch adds support for submitting map stages in a DAG individually so that we can make downstream decisions after seeing statistics about their output, as part of SPARK-9850. I also added more comments to many of the key classes in DAGScheduler. By itself, the patch is not super useful except maybe to switch between a shuffle and broadcast join, but with the other subtasks of SPARK-9850 we'll be able to do more interesting decisions.\r\n\r\nThe main entry point is SparkContext.submitMapStage, which lets you run a map stage and see stats about the map output sizes. Other stats could also be collected through accumulators. See AdaptiveSchedulingSuite for a short example."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/130871144","html_url":"https://github.com/apache/spark/pull/8180#issuecomment-130871144","issue_url":"https://api.github.com/repos/apache/spark/issues/8180","id":130871144,"user":{"login":"mateiz","id":228859,"avatar_url":"https://avatars.githubusercontent.com/u/228859?v=3","gravatar_id":"","url":"https://api.github.com/users/mateiz","html_url":"https://github.com/mateiz","followers_url":"https://api.github.com/users/mateiz/followers","following_url":"https://api.github.com/users/mateiz/following{/other_user}","gists_url":"https://api.github.com/users/mateiz/gists{/gist_id}","starred_url":"https://api.github.com/users/mateiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mateiz/subscriptions","organizations_url":"https://api.github.com/users/mateiz/orgs","repos_url":"https://api.github.com/users/mateiz/repos","events_url":"https://api.github.com/users/mateiz/events{/privacy}","received_events_url":"https://api.github.com/users/mateiz/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T22:51:10Z","updated_at":"2015-08-13T22:51:10Z","body":"retest this please"}}