{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4142","labels_url":"https://api.github.com/repos/apache/spark/issues/4142/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4142/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4142/events","html_url":"https://github.com/apache/spark/pull/4142","id":55067755,"number":4142,"title":"[SPARK-2669] [yarn] Distribute client configuration to AM.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-01-21T19:48:10Z","updated_at":"2015-02-12T01:18:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4142","html_url":"https://github.com/apache/spark/pull/4142","diff_url":"https://github.com/apache/spark/pull/4142.diff","patch_url":"https://github.com/apache/spark/pull/4142.patch"},"body":"Currently, when Spark launches the Yarn AM, the process will use\r\nthe local Hadoop configuration on the node where the AM launches,\r\nif one is present. A more correct approach is to use the same\r\nconfiguration used to launch the Spark job, since the user may\r\nhave made modifications (such as adding app-specific configs).\r\n\r\nThe approach taken here is to use the distributed cache to make\r\nall files in the Hadoop configuration directory available to the\r\nAM. This is a little overkill since only the AM needs them (the\r\nexecutors use the broadcast Hadoop configuration from the driver),\r\nbut is the easier approach.\r\n\r\nEven though only a few files in that directory may end up being\r\nused, all of them are uploaded. This allows supporting use cases\r\nsuch as when auxiliary configuration files are used for SSL\r\nconfiguration, or when uploading a Hive configuration directory.\r\nNot all of these may be reflected in a o.a.h.conf.Configuration object,\r\nbut may be needed when a driver in cluster mode instantiates, for\r\nexample, a HiveConf object instead."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/74000190","html_url":"https://github.com/apache/spark/pull/4142#issuecomment-74000190","issue_url":"https://api.github.com/repos/apache/spark/issues/4142","id":74000190,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-02-12T01:18:51Z","updated_at":"2015-02-12T01:18:51Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27310/\nTest FAILed."}}