{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/33484523","id":33484523,"diff_hunk":"@@ -256,6 +256,157 @@ still required, though there is only one application available.  Eg. to see the\n running app, you would go to `http://localhost:4040/api/v1/applications/[app-id]/jobs`.  This is to\n keep the paths consistent in both modes.\n \n+## Hadoop YARN Timeline service history provider\n+\n+As well as the Filesystem History Provider, Spark can integrate with the Hadoop YARN \n+\"Application Timeline Service\". This is a service which runs in a YARN cluster, recording\n+application- and YARN- published events to a database, retrieving them on request.\n+\n+Spark integrates with the timeline service by\n+1. Publishing events to the timeline service as applications execute.\n+2. Listing application histories published to the timeline service.\n+3. Retrieving the details of specific application histories.\n+\n+### Configuring the Timeline Service\n+\n+For details on configuring and starting the timeline service, consult the Hadoop documentation.\n+\n+From the perspective of Spark, the key requirements are\n+1. The YARN timeline service must be running.\n+1. Its URL is known, and configured in the `yarn-site.xml` configuration file.","path":"docs/monitoring.md","position":32,"original_position":32,"commit_id":"6aea1b6cbc91af47954d019e8c134a5342f4bba7","original_commit_id":"6aea1b6cbc91af47954d019e8c134a5342f4bba7","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"body":"Nice to know, but it would be good for the source file to be consistent.","created_at":"2015-06-29T16:46:49Z","updated_at":"2015-06-29T16:46:49Z","html_url":"https://github.com/apache/spark/pull/5423#discussion_r33484523","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/5423","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/33484523"},"html":{"href":"https://github.com/apache/spark/pull/5423#discussion_r33484523"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/5423"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5423","id":32861761,"html_url":"https://github.com/apache/spark/pull/5423","diff_url":"https://github.com/apache/spark/pull/5423.diff","patch_url":"https://github.com/apache/spark/pull/5423.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/5423","number":5423,"state":"open","locked":false,"title":"SPARK-1537 [WiP] Application Timeline Server integration","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"body":"This a snapshot of the work in progress. It's a superset of zhzhan's work, compiling against the master branch and with a lot more tests.\r\n\r\n1. there's now a hadoop-2.6 profile in the root pom; this is used to include the yarn/history source tree containing the new sources and tests\r\n1. As well as Marcelo's mock tests (which are all working happily), there's now some functional tests which use the timeline client to marshall a spark event, push it to an in-VM timeline server, then verify it's there/unmarshallable.\r\n1. There's some improved robustness in the production-side code.\r\n\r\nThis is not complete; i've pushed it up to show the current status. \r\n\r\nWhat's needed now is the REST operations to retrieve data, tests that everything wired up works.\r\n\r\n## Production Code\r\n\r\n1. I'm going to add a Jersey client to talk to ATS, avoiding any use of private hadoop classes, and resulting in code that works with Hadoop 2.6+\r\n1. I think the code to do PUT operations needs to be made async, so transient ATS outages don't block things. It's already queued & running in its own thread, but its posting in a synchronized section.\r\n\r\n## Tests\r\n\r\n* listening to spark context -> posted events in the ATS\r\n* retrieving events from the ATS via GET\r\n* joined up history provider\r\n\r\n## Hadoop side\r\n\r\nA statement of the compatibility policy of the current ATS read and write REST API","created_at":"2015-04-08T13:52:10Z","updated_at":"2015-06-29T16:46:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1175b98219c0b37a512e507b364734c8fabcaec0","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/5423/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/5423/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/5423/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/6aea1b6cbc91af47954d019e8c134a5342f4bba7","head":{"label":"steveloughran:stevel/feature/SPARK-1537-ATS","ref":"stevel/feature/SPARK-1537-ATS","sha":"6aea1b6cbc91af47954d019e8c134a5342f4bba7","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"repo":{"id":32387210,"name":"spark","full_name":"steveloughran/spark","owner":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/steveloughran/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/steveloughran/spark","forks_url":"https://api.github.com/repos/steveloughran/spark/forks","keys_url":"https://api.github.com/repos/steveloughran/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/steveloughran/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/steveloughran/spark/teams","hooks_url":"https://api.github.com/repos/steveloughran/spark/hooks","issue_events_url":"https://api.github.com/repos/steveloughran/spark/issues/events{/number}","events_url":"https://api.github.com/repos/steveloughran/spark/events","assignees_url":"https://api.github.com/repos/steveloughran/spark/assignees{/user}","branches_url":"https://api.github.com/repos/steveloughran/spark/branches{/branch}","tags_url":"https://api.github.com/repos/steveloughran/spark/tags","blobs_url":"https://api.github.com/repos/steveloughran/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/steveloughran/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/steveloughran/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/steveloughran/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/steveloughran/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/steveloughran/spark/languages","stargazers_url":"https://api.github.com/repos/steveloughran/spark/stargazers","contributors_url":"https://api.github.com/repos/steveloughran/spark/contributors","subscribers_url":"https://api.github.com/repos/steveloughran/spark/subscribers","subscription_url":"https://api.github.com/repos/steveloughran/spark/subscription","commits_url":"https://api.github.com/repos/steveloughran/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/steveloughran/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/steveloughran/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/steveloughran/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/steveloughran/spark/contents/{+path}","compare_url":"https://api.github.com/repos/steveloughran/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/steveloughran/spark/merges","archive_url":"https://api.github.com/repos/steveloughran/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/steveloughran/spark/downloads","issues_url":"https://api.github.com/repos/steveloughran/spark/issues{/number}","pulls_url":"https://api.github.com/repos/steveloughran/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/steveloughran/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/steveloughran/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/steveloughran/spark/labels{/name}","releases_url":"https://api.github.com/repos/steveloughran/spark/releases{/id}","created_at":"2015-03-17T10:36:33Z","updated_at":"2015-03-17T10:36:52Z","pushed_at":"2015-06-26T14:02:27Z","git_url":"git://github.com/steveloughran/spark.git","ssh_url":"git@github.com:steveloughran/spark.git","clone_url":"https://github.com/steveloughran/spark.git","svn_url":"https://github.com/steveloughran/spark","homepage":null,"size":109055,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"bbdfc0a40fb39760c122e7b9ce80aa1e340e55ee","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-06-29T16:03:18Z","pushed_at":"2015-06-29T16:44:27Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1531523,"stargazers_count":4575,"watchers_count":4575,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":4033,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":382,"forks":4033,"open_issues":382,"watchers":4575,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/5423"},"html":{"href":"https://github.com/apache/spark/pull/5423"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/5423"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/5423/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/5423/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/5423/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/6aea1b6cbc91af47954d019e8c134a5342f4bba7"}}}}