{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5423","labels_url":"https://api.github.com/repos/apache/spark/issues/5423/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5423/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5423/events","html_url":"https://github.com/apache/spark/pull/5423","id":67135385,"number":5423,"title":"SPARK-1537 [WiP] Application Timeline Server integration","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":87,"created_at":"2015-04-08T13:52:10Z","updated_at":"2015-10-26T17:37:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5423","html_url":"https://github.com/apache/spark/pull/5423","diff_url":"https://github.com/apache/spark/pull/5423.diff","patch_url":"https://github.com/apache/spark/pull/5423.patch"},"body":"This a snapshot of the work in progress. It's a superset of zhzhan's work, compiling against the master branch and with a lot more tests.\r\n\r\n1. there's now a hadoop-2.6 profile in the root pom; this is used to include the yarn/history source tree containing the new sources and tests\r\n1. As well as Marcelo's mock tests (which are all working happily), there's now some functional tests which use the timeline client to marshall a spark event, push it to an in-VM timeline server, then verify it's there/unmarshallable.\r\n1. There's some improved robustness in the production-side code.\r\n\r\nThis is not complete; i've pushed it up to show the current status. \r\n\r\nWhat's needed now is the REST operations to retrieve data, tests that everything wired up works.\r\n\r\n## Production Code\r\n\r\n1. I'm going to add a Jersey client to talk to ATS, avoiding any use of private hadoop classes, and resulting in code that works with Hadoop 2.6+\r\n1. I think the code to do PUT operations needs to be made async, so transient ATS outages don't block things. It's already queued & running in its own thread, but its posting in a synchronized section.\r\n\r\n## Tests\r\n\r\n* listening to spark context -> posted events in the ATS\r\n* retrieving events from the ATS via GET\r\n* joined up history provider\r\n\r\n## Hadoop side\r\n\r\nA statement of the compatibility policy of the current ATS read and write REST API"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/151220020","html_url":"https://github.com/apache/spark/pull/5423#issuecomment-151220020","issue_url":"https://api.github.com/repos/apache/spark/issues/5423","id":151220020,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T17:37:17Z","updated_at":"2015-10-26T17:37:17Z","body":" Merged build triggered."}}