{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4688","labels_url":"https://api.github.com/repos/apache/spark/issues/4688/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4688/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4688/events","html_url":"https://github.com/apache/spark/pull/4688","id":58174225,"number":4688,"title":"[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS","user":{"login":"harishreedharan","id":1481194,"avatar_url":"https://avatars.githubusercontent.com/u/1481194?v=3","gravatar_id":"","url":"https://api.github.com/users/harishreedharan","html_url":"https://github.com/harishreedharan","followers_url":"https://api.github.com/users/harishreedharan/followers","following_url":"https://api.github.com/users/harishreedharan/following{/other_user}","gists_url":"https://api.github.com/users/harishreedharan/gists{/gist_id}","starred_url":"https://api.github.com/users/harishreedharan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harishreedharan/subscriptions","organizations_url":"https://api.github.com/users/harishreedharan/orgs","repos_url":"https://api.github.com/users/harishreedharan/repos","events_url":"https://api.github.com/users/harishreedharan/events{/privacy}","received_events_url":"https://api.github.com/users/harishreedharan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":164,"created_at":"2015-02-19T06:50:49Z","updated_at":"2015-04-29T03:15:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4688","html_url":"https://github.com/apache/spark/pull/4688","diff_url":"https://github.com/apache/spark/pull/4688.diff","patch_url":"https://github.com/apache/spark/pull/4688.patch"},"body":"Current Spark apps running on Secure YARN/HDFS would not be able to write data\r\nto HDFS after 7 days, since delegation tokens cannot be renewed beyond that. This\r\nmeans Spark Streaming apps will not be able to run on Secure YARN.\r\n\r\nThis commit adds basic functionality to fix this issue. In this patch:\r\n- new parameters are added - principal and keytab, which can be used to login to a KDC\r\n- the client logs in, and then get tokens to start the AM\r\n- the keytab is copied to the staging directory\r\n- the AM waits for 60% of the time till expiry of the tokens and then logs in using the keytab\r\n- each time after 60% of the time, new tokens are created and sent to the executors\r\n\r\nCurrently, to avoid complicating the architecture, we set the keytab and principal in the\r\nSparkHadoopUtil singleton, and schedule a login. Once the login is completed, a callback is scheduled.\r\n\r\nThis is being posted for feedback, so I can gather feedback on the general implementation.\r\n\r\nThere are currently a bunch of things to do:\r\n- [x] logging\r\n- [x] testing - I plan to manually test this soon. If you have ideas of how to add unit tests, comment.\r\n- [x] add code to ensure that if these params are set in non-YARN cluster mode, we complain\r\n- [x] documentation \r\n- [x] Have the executors request for credentials from the AM, so that retries are possible."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/97291369","html_url":"https://github.com/apache/spark/pull/4688#issuecomment-97291369","issue_url":"https://api.github.com/repos/apache/spark/issues/4688","id":97291369,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T03:15:39Z","updated_at":"2015-04-29T03:15:39Z","body":"  [Test build #31203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31203/consoleFull) for   PR 4688 at commit [`09fe224`](https://github.com/apache/spark/commit/09fe2244f82b5037772c40693378b2def43b12bb).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds no public classes.\n * This patch does not change any dependencies."}}