{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9315","labels_url":"https://api.github.com/repos/apache/spark/issues/9315/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9315/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9315/events","html_url":"https://github.com/apache/spark/pull/9315","id":113730529,"number":9315,"title":"[SPARK-11361][Streaming] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-10-28T02:31:45Z","updated_at":"2015-11-10T22:52:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9315","html_url":"https://github.com/apache/spark/pull/9315","diff_url":"https://github.com/apache/spark/pull/9315.diff","patch_url":"https://github.com/apache/spark/pull/9315.patch"},"body":"Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed to be overridden by the RDD operations. So in case of `DStream.foreachRDD`, all the RDDs generated inside the foreachRDD get the same scope - `foreachRDD @ <time>`, as set by the `ForeachDStream`. So it is hard to debug generated RDDs in the RDD DAG viz in the Spark UI. \r\n\r\nThis patch allows the RDD operations inside `DStream.transform` and `DStream.foreachRDD` to append their own scopes to the earlier DStream scope. \r\n\r\nI have also slightly tweaked how callsites are set such that the short callsite reflects the RDD operation name and line number. This tweak is necessary as callsites are not managed through scopes (which support nesting and overriding) and I didnt want to add another local property to control nesting and overriding of callsites.\r\n\r\n## Before:\r\n![image](https://cloud.githubusercontent.com/assets/663212/10808548/fa71c0c4-7da9-11e5-9af0-5737793a146f.png)\r\n\r\n## After:\r\n![image](https://cloud.githubusercontent.com/assets/663212/10808659/37bc45b6-7dab-11e5-8041-c20be6a9bc26.png)\r\n\r\nThe code that was used to generate this is: \r\n```\r\n    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)\r\n    val words = lines.flatMap(_.split(\" \"))\r\n    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)\r\n    wordCounts.foreachRDD { rdd =>\r\n      val temp = rdd.map { _ -> 1 }.reduceByKey( _ + _)\r\n      val temp2 = temp.map { _ -> 1}.reduceByKey(_ + _)\r\n      val count = temp2.count\r\n      println(count)\r\n    }\r\n```\r\n\r\nNote\r\n- The inner scopes of the RDD operations map/reduceByKey inside foreachRDD is visible\r\n- The short callsites of stages refers to the line number of the RDD ops rather than the same line number of foreachRDD in all three cases."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/155593553","html_url":"https://github.com/apache/spark/pull/9315#issuecomment-155593553","issue_url":"https://api.github.com/repos/apache/spark/issues/9315","id":155593553,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T22:52:41Z","updated_at":"2015-11-10T22:52:41Z","body":"**[Test build #45564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45564/consoleFull)** for PR 9315 at commit [`955658a`](https://github.com/apache/spark/commit/955658a02ee6e05da6c783f5230ad8f0b80c2489)."}}