{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/29640742","id":29640742,"diff_hunk":"@@ -174,6 +174,79 @@ making it easy to identify slow tasks, data skew, etc.\n \n Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI.\n \n+## REST API\n+\n+In addition to viewing the metrics in the UI, they are also available as JSON.  This gives developers\n+an easy way to create new visualizations and monitoring tools for Spark.  The JSON is available for\n+both running applications, an in the history server.  The endpoints are mounted at `/json/v1`.  Eg.,\n+for the history server, they would typically be accessible at `http://<server-url>:18080/json/v1`.\n+\n+<table class=\"table\">\n+  <tr><th>Endpoint</th><th>Meaning</th></tr>\n+  <tr>\n+    <td><code>/applications</code></td>\n+    <td>A list of all applications</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/jobs</code></td>\n+    <td>A list of all jobs for a given application</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/jobs/[job-id]</code></td>\n+    <td>Details for one job</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/stages</code></td>\n+    <td>A list of all stages for a given application</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/stages/[stage-id]</code></td>\n+    <td>A list of all attempts for a given stage</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]</code></td>\n+    <td>Details for the given stage attempt</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary</code></td>\n+    <td>Summary metrics of all tasks in a stage attempt</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList</code></td>\n+    <td>A list of all tasks for a given stage attempt</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/executors</code></td>\n+    <td>A list of all executors for the given application</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/storage/rdd</code></td>\n+    <td>A list of stored RDDs for the given application</td>\n+  </tr>\n+  <tr>\n+    <td><code>/applications/[app-id]/storage/rdd/[rdd-id]</code></td>\n+    <td>Details for the storage status of a given RDD</td>\n+  </tr>\n+</table>\n+\n+When running on Yarn, each application has multiple attempts, so `[app-id]` is actually\n+`[app-id]/[attempt-id]` in all cases.\n+\n+These endpoints have been strongly versioned to make it easier to develop applications on top.\n+ In particular, Spark guarantees:\n+\n+* Endpoints will never be removed from one version\n+* Individual fields will never be removed for any given endpoint\n+* New endpoints may be added\n+* New fields may be added to existing endpoints\n+* New versions of the api may be added in the future at a separate endpoint (eg., `json/v2`).  New versions are *not* required to be backwards compatible.\n+* Api versions may be dropped, but only after at least one minor release of existing beside a new api version","path":"docs/monitoring.md","position":70,"original_position":70,"commit_id":"5e78b4fd07d844d5c7a4b86519bda7f0530dc03f","original_commit_id":"5e78b4fd07d844d5c7a4b86519bda7f0530dc03f","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"body":"I wasn't sure if we want to make a stronger guarantee than this ... eg. maybe we should keep `json/v1` around as long as we're on Spark 1.x.  But I figured I'd word it like this for now to give us more room with the initial version.  Definitely open to other opinions","created_at":"2015-05-05T02:26:04Z","updated_at":"2015-05-05T02:26:04Z","html_url":"https://github.com/apache/spark/pull/4435#discussion_r29640742","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/4435","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/29640742"},"html":{"href":"https://github.com/apache/spark/pull/4435#discussion_r29640742"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/4435"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4435","id":28843992,"html_url":"https://github.com/apache/spark/pull/4435","diff_url":"https://github.com/apache/spark/pull/4435.diff","patch_url":"https://github.com/apache/spark/pull/4435.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/4435","number":4435,"state":"open","locked":false,"title":"[SPARK-3454] separate json endpoints for data in the UI","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"body":"Exposes data available in the UI as json over http.  Key points:\r\n\r\n* new endpoints, handled independently of existing XyzPage classes.  Root entrypoint is `JsonRootResource`\r\n* Uses jersey + jackson for routing & converting POJOs into json\r\n* tests against known results in `HistoryServerSuite`\r\n* also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages.","created_at":"2015-02-06T21:35:40Z","updated_at":"2015-05-05T02:26:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c3930c74f296fd3b9a6ad293d132d95c238a3f66","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/4435/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/4435/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/4435/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/5e78b4fd07d844d5c7a4b86519bda7f0530dc03f","head":{"label":"squito:SPARK-3454","ref":"SPARK-3454","sha":"5e78b4fd07d844d5c7a4b86519bda7f0530dc03f","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"repo":{"id":29212749,"name":"spark","full_name":"squito/spark","owner":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/squito/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/squito/spark","forks_url":"https://api.github.com/repos/squito/spark/forks","keys_url":"https://api.github.com/repos/squito/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/squito/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/squito/spark/teams","hooks_url":"https://api.github.com/repos/squito/spark/hooks","issue_events_url":"https://api.github.com/repos/squito/spark/issues/events{/number}","events_url":"https://api.github.com/repos/squito/spark/events","assignees_url":"https://api.github.com/repos/squito/spark/assignees{/user}","branches_url":"https://api.github.com/repos/squito/spark/branches{/branch}","tags_url":"https://api.github.com/repos/squito/spark/tags","blobs_url":"https://api.github.com/repos/squito/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/squito/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/squito/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/squito/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/squito/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/squito/spark/languages","stargazers_url":"https://api.github.com/repos/squito/spark/stargazers","contributors_url":"https://api.github.com/repos/squito/spark/contributors","subscribers_url":"https://api.github.com/repos/squito/spark/subscribers","subscription_url":"https://api.github.com/repos/squito/spark/subscription","commits_url":"https://api.github.com/repos/squito/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/squito/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/squito/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/squito/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/squito/spark/contents/{+path}","compare_url":"https://api.github.com/repos/squito/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/squito/spark/merges","archive_url":"https://api.github.com/repos/squito/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/squito/spark/downloads","issues_url":"https://api.github.com/repos/squito/spark/issues{/number}","pulls_url":"https://api.github.com/repos/squito/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/squito/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/squito/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/squito/spark/labels{/name}","releases_url":"https://api.github.com/repos/squito/spark/releases{/id}","created_at":"2015-01-13T21:43:10Z","updated_at":"2015-01-13T21:43:26Z","pushed_at":"2015-05-05T02:17:19Z","git_url":"git://github.com/squito/spark.git","ssh_url":"git@github.com:squito/spark.git","clone_url":"https://github.com/squito/spark.git","svn_url":"https://github.com/squito/spark","homepage":null,"size":102992,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"8aa5aea7fee0ae9cd34e16c30655ee02b8747455","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-05-05T01:54:06Z","pushed_at":"2015-05-05T02:17:21Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1590090,"stargazers_count":3856,"watchers_count":3856,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":3346,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":308,"forks":3346,"open_issues":308,"watchers":3856,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/4435"},"html":{"href":"https://github.com/apache/spark/pull/4435"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/4435"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/4435/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/4435/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/4435/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/5e78b4fd07d844d5c7a4b86519bda7f0530dc03f"}}}}