{"action":"created","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/38","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/38/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/38/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/38","id":53454342,"number":38,"title":"Unable to use yarn-client as master","user":{"login":"chrismcbride","id":662229,"avatar_url":"https://avatars.githubusercontent.com/u/662229?v=3","gravatar_id":"","url":"https://api.github.com/users/chrismcbride","html_url":"https://github.com/chrismcbride","followers_url":"https://api.github.com/users/chrismcbride/followers","following_url":"https://api.github.com/users/chrismcbride/following{/other_user}","gists_url":"https://api.github.com/users/chrismcbride/gists{/gist_id}","starred_url":"https://api.github.com/users/chrismcbride/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrismcbride/subscriptions","organizations_url":"https://api.github.com/users/chrismcbride/orgs","repos_url":"https://api.github.com/users/chrismcbride/repos","events_url":"https://api.github.com/users/chrismcbride/events{/privacy}","received_events_url":"https://api.github.com/users/chrismcbride/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-01-05T22:08:01Z","updated_at":"2015-03-04T19:12:11Z","closed_at":null,"body":"Setting master to 'yarn-client' in the config results in this error during startup:\r\n\r\n2015-01-05 22:06:55,765 ERROR [JobServer-akka.actor.default-dispatcher-5] actor.OneForOneStrategy (Slf4jLogger.scala:apply$mcV$sp(66)) - exception during creation\r\nakka.actor.ActorInitializationException: exception during creation\r\n\tat akka.actor.ActorInitializationException$.apply(Actor.scala:218)\r\n\tat akka.actor.ActorCell.create(ActorCell.scala:578)\r\n\tat akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)\r\n\tat akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)\r\n\tat akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:218)\r\n\tat akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:526)\r\n\tat akka.util.Reflect$.instantiate(Reflect.scala:65)\r\n\tat akka.actor.Props.newActor(Props.scala:337)\r\n\tat akka.actor.ActorCell.newActor(ActorCell.scala:534)\r\n\tat akka.actor.ActorCell.create(ActorCell.scala:560)\r\n\t... 9 more\r\nCaused by: java.lang.RuntimeException: Could not parse Master URL: 'yarn-client'\r\n\tat spark.jobserver.SparkWebUiActor.getSparkHostName(SparkWebUiActor.scala:85)\r\n\tat spark.jobserver.SparkWebUiActor.<init>(SparkWebUiActor.scala:40)\r\n\t... 17 more"},"comment":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/comments/77224320","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/38#issuecomment-77224320","issue_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/38","id":77224320,"user":{"login":"curran","id":68416,"avatar_url":"https://avatars.githubusercontent.com/u/68416?v=3","gravatar_id":"","url":"https://api.github.com/users/curran","html_url":"https://github.com/curran","followers_url":"https://api.github.com/users/curran/followers","following_url":"https://api.github.com/users/curran/following{/other_user}","gists_url":"https://api.github.com/users/curran/gists{/gist_id}","starred_url":"https://api.github.com/users/curran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/curran/subscriptions","organizations_url":"https://api.github.com/users/curran/orgs","repos_url":"https://api.github.com/users/curran/repos","events_url":"https://api.github.com/users/curran/events{/privacy}","received_events_url":"https://api.github.com/users/curran/received_events","type":"User","site_admin":false},"created_at":"2015-03-04T19:12:11Z","updated_at":"2015-03-04T19:12:11Z","body":"I am also not able to use yarn-client mode. I set `master = \"yarn-client\"` in `config/local.conf`, then launched the server with `reStart config/local.conf`.\r\n\r\nThe jobserver accepted the example jar with `curl -data-binary @job-server-tests/target/job-server-tests-0.5.0.jar localhost:8090/jars/test`, but when I try to run the example I get the following error:\r\n\r\n```\r\n$ curl d \"input.string = a b c a b see\" 'localhost:8090/jobs?appName=test&classPath=spark.jobserver.WordCountExample''\r\n\r\n{ \"status\": \"ERROR\", \"result\": { \"errorClass\": \"java.lang.RuntimeException\", \"cause\": null, \"stack\":\r\n[\"org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1784)\",\r\n\"org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105)\",\r\n\"org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:180)\",\r\n\"org.apache.spark.SparkEnv$.create(SparkEnv.scala:292)\",\r\n\"org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)\",\r\n\"org.apache.spark.SparkContext.<init>(SparkContext.scala:232)\",\r\n\"spark.jobserver.context.DefaultSparkContextFactory$$anon$1.<init>(SparkContextFactory.scala:41)\",\r\n\"spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:41)\",\r\n\"spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:34)\",\r\n\"spark.jobserver.JobManagerActor.createContextFromConfig(JobManagerActor.scala:261)\",\r\n\"spark.jobserver.JobManagerActor$$anonfun$wrappedReceive$1.applyOrElse(JobManagerActor.scala:104)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\",\r\n\"ooyala.common.akka.ActorStack$$anonfun$receive$1.applyOrElse(ActorStack.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\",\r\n\"ooyala.common.akka.Slf4jLogging$$anonfun$receive$1$$anonfun$applyOrElse$1.apply$mcV$sp(Slf4jLogging.scala:26)\",\r\n\"ooyala.common.akka.Slf4jLogging$class.ooyala$common$akka$Slf4jLogging$$withAkkaSourceLogging(Slf4jLogging.scala:35)\",\r\n\"ooyala.common.akka.Slf4jLogging$$anonfun$receive$1.applyOrElse(Slf4jLogging.scala:25)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)\",\r\n\"scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)\",\r\n\"ooyala.common.akka.ActorMetrics$$anonfun$receive$1.applyOrElse(ActorMetrics.scala:24)\",\r\n\"akka.actor.Actor$class.aroundReceive(Actor.scala:465)\",\r\n\"ooyala.common.akka.InstrumentedActor.aroundReceive(InstrumentedActor.scala:8)\",\r\n\"akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)\",\r\n\"akka.actor.ActorCell.invoke(ActorCell.scala:487)\",\r\n\"akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)\",\r\n\"akka.dispatch.Mailbox.run(Mailbox.scala:220)\",\r\n\"akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)\",\r\n\"scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\",\r\n\"scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\",\r\n\"scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\",\r\n\"scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\"],\r\n\"causingClass\": \"java.lang.ExceptionInInitializerError\", \"message\":\r\n\"java.lang.ExceptionInInitializerError\" } }\r\n```\r\nThe Word Count example works for me when `master = \"local[4]\"`, but gives this error when `master = \"yarn-client\"`.\r\n\r\nI have the following environment variables set, but I'm not sure which are being used by spark-jobserver: HADOOP_CONF_DIR, YARN_CONF_DIR, SPARK_HOME, SPARK_CONF_DIR. I am on a Mac, using Apache Hadoop-2.6.0 and Spark-1.2.1.\r\n\r\nAny ideas on what is causing this error, or advice for what to try next? Thank you."}}