{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7774","labels_url":"https://api.github.com/repos/apache/spark/issues/7774/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7774/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7774/events","html_url":"https://github.com/apache/spark/pull/7774","id":98081240,"number":7774,"title":"[SPARK-8861][SPARK-8862][SQL] Add basic instrumentation to each SparkPlan operator and add a new SQL tab","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":58,"created_at":"2015-07-30T04:02:37Z","updated_at":"2015-08-04T02:58:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7774","html_url":"https://github.com/apache/spark/pull/7774","diff_url":"https://github.com/apache/spark/pull/7774.diff","patch_url":"https://github.com/apache/spark/pull/7774.patch"},"body":"This PR includes the following changes:\r\n\r\n### SPARK-8862: Add basic instrumentation to each SparkPlan operator\r\n\r\nA SparkPlan can override `def accumulators: Map[String, Accumulator[_]]` to expose its metrics that can be displayed in UI. The UI will use them to track the updates and show them in the web page in real-time.\r\n\r\n### SparkSQLExecution and SQLSparkListener\r\n\r\n`SparkSQLExecution.withNewExecutionId` will set `spark.sql.execution.id` to the local properties so that we can use it to track all jobs that belong to the same query.\r\n \r\nSQLSparkListener is a listener to track all accumulator updates of all tasks for a query. It receives them from heartbeats can the UI can query them in real-time.\r\n\r\nWhen running a query, `SQLSparkListener.onExecutionStart` will be called. When a query is finished,  `SQLSparkListener.onExecutionEnd` will be called. And the Spark jobs with the same execution id will be tracked and stored with this query.\r\n\r\n`SQLSparkListener` has to store all accumulator updates for tasks separately. When a task fails and starts to retry, we need to drop the old accumulator updates. Because we can not revert our changes to an accumulator, we have to maintain these accumulator updates by ourselves so as to drop accumulator updates for a failed task. \r\n\r\n### SPARK-8862: A new SQL tab\r\nIncludes two pages: \r\n#### A page for all DataFrame/SQL queries \r\nIt will show the running, completed and failed queries in 3 tables. It also displays the jobs and their links for a query in each row.\r\n#### A detail page for a DataFrame/SQL query\r\nIn this page, it also shows the SparkPlan metrics in real-time. Run a long-running query, such as\r\n```\r\nval testData = sc.parallelize((1 to 1000000).map(i => (i, i.toString))).toDF()\r\ntestData.select($\"_1\").filter($\"_1\" < 1000).foreach(_ => Thread.sleep(60))\r\n```\r\nand you will see the metrics keep updating in real-time.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/7774)\n<!-- Reviewable:end -->\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127461480","html_url":"https://github.com/apache/spark/pull/7774#issuecomment-127461480","issue_url":"https://api.github.com/repos/apache/spark/issues/7774","id":127461480,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T02:58:06Z","updated_at":"2015-08-04T02:58:06Z","body":"  [Test build #39644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39644/consoleFull) for   PR 7774 at commit [`7ab4816`](https://github.com/apache/spark/commit/7ab48165011f1701ca1d22d6f616bec14f5f96cd)."}}