{"action":"created","issue":{"url":"https://api.github.com/repos/cloudera/hue/issues/244","labels_url":"https://api.github.com/repos/cloudera/hue/issues/244/labels{/name}","comments_url":"https://api.github.com/repos/cloudera/hue/issues/244/comments","events_url":"https://api.github.com/repos/cloudera/hue/issues/244/events","html_url":"https://github.com/cloudera/hue/issues/244","id":110416037,"number":244,"title":"[Livy] Spark submit paramaters - HDP","user":{"login":"syepes","id":333081,"avatar_url":"https://avatars.githubusercontent.com/u/333081?v=3","gravatar_id":"","url":"https://api.github.com/users/syepes","html_url":"https://github.com/syepes","followers_url":"https://api.github.com/users/syepes/followers","following_url":"https://api.github.com/users/syepes/following{/other_user}","gists_url":"https://api.github.com/users/syepes/gists{/gist_id}","starred_url":"https://api.github.com/users/syepes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/syepes/subscriptions","organizations_url":"https://api.github.com/users/syepes/orgs","repos_url":"https://api.github.com/users/syepes/repos","events_url":"https://api.github.com/users/syepes/events{/privacy}","received_events_url":"https://api.github.com/users/syepes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"erickt","id":84711,"avatar_url":"https://avatars.githubusercontent.com/u/84711?v=3","gravatar_id":"","url":"https://api.github.com/users/erickt","html_url":"https://github.com/erickt","followers_url":"https://api.github.com/users/erickt/followers","following_url":"https://api.github.com/users/erickt/following{/other_user}","gists_url":"https://api.github.com/users/erickt/gists{/gist_id}","starred_url":"https://api.github.com/users/erickt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erickt/subscriptions","organizations_url":"https://api.github.com/users/erickt/orgs","repos_url":"https://api.github.com/users/erickt/repos","events_url":"https://api.github.com/users/erickt/events{/privacy}","received_events_url":"https://api.github.com/users/erickt/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2015-10-08T10:17:00Z","updated_at":"2015-10-10T04:15:46Z","closed_at":null,"body":"Hello,\r\n\r\nI am trying to use Hue (7fc1bb4ed0) Spark Notebooks feature in our HDP environment, but the Livy server can not submit Spark jobs correctly to YARN as in HDP we need to pass the parameter java option \"hdp.version\".\r\n\r\nDoes there exist anyway to configure the Livy server so that is passes the options \"spark.*.extraJavaOptions\" when submitting a job?\r\n\r\nThese settings work with other tools like Zeppelin and Spark-shell:\r\n\r\nspark-defaults.conf\r\n```\r\nspark.driver.extraJavaOptions                -Dhdp.version=2.3.2.0-2950\r\nspark.executor.extraJavaOptions              -Dhdp.version=2.3.2.0-2950\r\nspark.yarn.am.extraJavaOptions               -Dhdp.version=2.3.2.0-2950\r\n```\r\nexport HADOOP_CONF_DIR=/etc/hadoop/conf\r\nexport SPARK_HOME=/opt/spark\r\n\r\n\r\nBest regards,\r\nSebastian"},"comment":{"url":"https://api.github.com/repos/cloudera/hue/issues/comments/147036124","html_url":"https://github.com/cloudera/hue/issues/244#issuecomment-147036124","issue_url":"https://api.github.com/repos/cloudera/hue/issues/244","id":147036124,"user":{"login":"erickt","id":84711,"avatar_url":"https://avatars.githubusercontent.com/u/84711?v=3","gravatar_id":"","url":"https://api.github.com/users/erickt","html_url":"https://github.com/erickt","followers_url":"https://api.github.com/users/erickt/followers","following_url":"https://api.github.com/users/erickt/following{/other_user}","gists_url":"https://api.github.com/users/erickt/gists{/gist_id}","starred_url":"https://api.github.com/users/erickt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erickt/subscriptions","organizations_url":"https://api.github.com/users/erickt/orgs","repos_url":"https://api.github.com/users/erickt/repos","events_url":"https://api.github.com/users/erickt/events{/privacy}","received_events_url":"https://api.github.com/users/erickt/received_events","type":"User","site_admin":false},"created_at":"2015-10-10T04:15:46Z","updated_at":"2015-10-10T04:15:46Z","body":"I know what's going on. The problem is both the livy and spark config file filter out all options that either start with `livy.` or `spark.`. Since this `hdp.version` doesn't start with either, it's not getting passed along. Livy does have some mechanisms to set this though. You could try running:\r\n\r\n```\r\nenv LIVY_REPL_JAVA_OPTS=\"-Dhdp.version=2.3.2.0-2950\" ./bin/livy-server\r\n```\r\n\r\nWhich should set the `--driver-java-options` for you. That should tide you over until we get a proper config value for this situation."}}