{"action":"created","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/310","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/310/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/310/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/310/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/310","id":116196439,"number":310,"title":"Job launch fails if Spark runs on Secure YARN Cluster","user":{"login":"mvle","id":9408668,"avatar_url":"https://avatars.githubusercontent.com/u/9408668?v=3","gravatar_id":"","url":"https://api.github.com/users/mvle","html_url":"https://github.com/mvle","followers_url":"https://api.github.com/users/mvle/followers","following_url":"https://api.github.com/users/mvle/following{/other_user}","gists_url":"https://api.github.com/users/mvle/gists{/gist_id}","starred_url":"https://api.github.com/users/mvle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mvle/subscriptions","organizations_url":"https://api.github.com/users/mvle/orgs","repos_url":"https://api.github.com/users/mvle/repos","events_url":"https://api.github.com/users/mvle/events{/privacy}","received_events_url":"https://api.github.com/users/mvle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-11-10T20:36:52Z","updated_at":"2015-11-18T23:10:16Z","closed_at":null,"body":"Hi,\r\n\r\nI've deployed a Secure YARN cluster with HDFS encryption and am trying to use the SJS for submitting Spark jobs.\r\n\r\nWhen submitting new jobs through the API, the job submission will fail the second time a job is submitted. Everything works fine if I create a long lasting job context and submit jobs to that single context. However, it looks like when SJS has to create a new context, the new context has a problem getting a delegation token. If I restart the SJS, then I can once again execute the job once, but the second time the job is submitted (a new context is created), it will fail once again.\r\n\r\nI think this might be related to how the spark context is destroyed and created, causing something to go out of sync in terms of caching tokens with the YARN/HDFS cluster. Does anyone have any experience with this or can give some clues how this can be fixed?\r\n\r\nI include error logs are below:\r\n\r\n{\r\n  \"status\": \"ERROR\",\r\n  \"result\": {\r\n    \"errorClass\": \"java.lang.RuntimeException\",\r\n    \"cause\": \"token (HDFS_DELEGATION_TOKEN token 42 for hadoop) can't be found in cache\",\r\n    \"stack\": [\"org.apache.hadoop.ipc.Client.call(Client.java:1476)\", \"org.apache.hadoop.ipc.Client.call(Client.java:1407)\", \"org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)\", \"com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)\", \"org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:933)\", \"sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\", \"sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\", \"sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\", \"java.lang.reflect.Method.invoke(Method.java:606)\", \"org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)\", \"org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)\", \"com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)\", \"org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1044)\", \"org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1543)\", \"org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:530)\", \"org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)\", \"org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2228)\", \"org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126)\", \"org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123)\", \"scala.collection.immutable.Set$Set1.foreach(Set.scala:74)\", \"org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes(YarnSparkHadoopUtil.scala:123)\", \"org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:495)\", \"org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:528)\", \"org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:628)\", \"org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)\", \"org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)\", \"org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)\", \"org.apache.spark.SparkContext.<init>(SparkContext.scala:523)\", \"spark.jobserver.context.DefaultSparkContextFactory$$anon$1.<init>(SparkContextFactory.scala:53)\", \"spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:53)\", \"spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:48)\", \"spark.jobserver.context.SparkContextFactory$class.makeContext(SparkContextFactory.scala:37)\", \"spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:48)\", \"spark.jobserver.JobManagerActor.createContextFromConfig(JobManagerActor.scala:287)\", \"spark.jobserver.JobManagerActor$$anonfun$wrappedReceive$1.applyOrElse(JobManagerActor.scala:109)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\", \"ooyala.common.akka.ActorStack$$anonfun$receive$1.applyOrElse(ActorStack.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\", \"ooyala.common.akka.Slf4jLogging$$anonfun$receive$1$$anonfun$applyOrElse$1.apply$mcV$sp(Slf4jLogging.scala:26)\", \"ooyala.common.akka.Slf4jLogging$class.ooyala$common$akka$Slf4jLogging$$withAkkaSourceLogging(Slf4jLogging.scala:35)\", \"ooyala.common.akka.Slf4jLogging$$anonfun$receive$1.applyOrElse(Slf4jLogging.scala:25)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\", \"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\", \"ooyala.common.akka.ActorMetrics$$anonfun$receive$1.applyOrElse(ActorMetrics.scala:24)\", \"akka.actor.Actor$class.aroundReceive(Actor.scala:467)\", \"ooyala.common.akka.InstrumentedActor.aroundReceive(InstrumentedActor.scala:8)\", \"akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)\", \"akka.actor.ActorCell.invoke(ActorCell.scala:487)\", \"akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)\", \"akka.dispatch.Mailbox.run(Mailbox.scala:220)\", \"akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)\", \"scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\", \"scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\", \"scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\", \"scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\"],\r\n    \"causingClass\": \"org.apache.hadoop.ipc.RemoteException\",\r\n    \"message\": \"org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 42 for hadoop) can't be found in cache\"\r\n  }"},"comment":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/comments/157894473","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/310#issuecomment-157894473","issue_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/310","id":157894473,"user":{"login":"velvia","id":1062875,"avatar_url":"https://avatars.githubusercontent.com/u/1062875?v=3","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"created_at":"2015-11-18T23:10:16Z","updated_at":"2015-11-18T23:10:16Z","body":"Actually, tests should be disabled when you build and deploy already.   Just follow the normal deploy instructions and you should be good to go.\n\n> On Nov 18, 2015, at 1:48 PM, Michael Le <notifications@github.com> wrote:\n> \n> Can you please enlighten me?\n> \n> —\n> Reply to this email directly or view it on GitHub <https://github.com/spark-jobserver/spark-jobserver/issues/310#issuecomment-157875549>.\n> \n\n"}}