{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5729","labels_url":"https://api.github.com/repos/apache/spark/issues/5729/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5729/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5729/events","html_url":"https://github.com/apache/spark/pull/5729","id":71433439,"number":5729,"title":"[SPARK-6943][WIP] DAG visualization on SparkUI","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":35,"created_at":"2015-04-27T23:55:47Z","updated_at":"2015-05-03T20:00:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5729","html_url":"https://github.com/apache/spark/pull/5729","diff_url":"https://github.com/apache/spark/pull/5729.diff","patch_url":"https://github.com/apache/spark/pull/5729.patch"},"body":"Screenshot is out of date. I will post an updated version shortly.\r\n-----------------------------------------------------------------------------------------------------------------------------------------\r\nThis patch adds the functionality to display the per-stage RDD DAG on the SparkUI. On a high level, the information displayed include:\r\n- The names and IDs of all RDDs involved in a stage\r\n- How these RDDs depend on each other, and\r\n- What scopes these RDDs are defined in\r\nScope here refers to the user-facing operation that created the RDDs (e.g. `textFile`, `treeAggregate`).\r\n\r\nThis blatantly stole a few lines of HTML and JavaScript from #5547 (thanks @shroffpradyumn!). We will have to deal with the merge conflicts a little later.\r\n\r\nFor instance, here's what the first stage of word count looks like:\r\n```\r\nsc.textFile(\"README.md\")\r\n  .flatMap { _.split(\" \") }\r\n  .map { (_, 1) }\r\n  .reduceByKey { _ + _ }\r\n  .sortBy(-_._2)\r\n```\r\n![](https://issues.apache.org/jira/secure/attachment/12728661/with-closures.png)"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/98532652","html_url":"https://github.com/apache/spark/pull/5729#issuecomment-98532652","issue_url":"https://api.github.com/repos/apache/spark/issues/5729","id":98532652,"user":{"login":"pwendell","id":320616,"avatar_url":"https://avatars.githubusercontent.com/u/320616?v=3","gravatar_id":"","url":"https://api.github.com/users/pwendell","html_url":"https://github.com/pwendell","followers_url":"https://api.github.com/users/pwendell/followers","following_url":"https://api.github.com/users/pwendell/following{/other_user}","gists_url":"https://api.github.com/users/pwendell/gists{/gist_id}","starred_url":"https://api.github.com/users/pwendell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwendell/subscriptions","organizations_url":"https://api.github.com/users/pwendell/orgs","repos_url":"https://api.github.com/users/pwendell/repos","events_url":"https://api.github.com/users/pwendell/events{/privacy}","received_events_url":"https://api.github.com/users/pwendell/received_events","type":"User","site_admin":false},"created_at":"2015-05-03T20:00:22Z","updated_at":"2015-05-03T20:00:22Z","body":"By the way - I think the answer to my question is simply that the general scope mechanism allows a scope to span multiple jobs or stages, but the current visualization will never plot a scope encapsulating a job, because it only plots jobs as the coarsest-grained concept."}}