{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/2982","labels_url":"https://api.github.com/repos/apache/spark/issues/2982/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/2982/comments","events_url":"https://api.github.com/repos/apache/spark/issues/2982/events","html_url":"https://github.com/apache/spark/pull/2982","id":47072713,"number":2982,"title":"[SPARK-4048] Enhance and extend hadoop-provided profile.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":74,"created_at":"2014-10-28T19:50:26Z","updated_at":"2015-01-06T18:13:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/2982","html_url":"https://github.com/apache/spark/pull/2982","diff_url":"https://github.com/apache/spark/pull/2982.diff","patch_url":"https://github.com/apache/spark/pull/2982.patch"},"body":"This change does a few things to make the hadoop-provided profile more useful:\r\n\r\n- Create new profiles for other libraries / services that might be provided by the infrastructure\r\n- Simplify and fix the poms so that the profiles are only activated while building assemblies.\r\n- Fix tests so that they're able to run when the profiles are activated\r\n- Add a new env variable to be used by distributions that use these profiles to provide the runtime\r\n  classpath for Spark jobs and daemons."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/68906274","html_url":"https://github.com/apache/spark/pull/2982#issuecomment-68906274","issue_url":"https://api.github.com/repos/apache/spark/issues/2982","id":68906274,"user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"created_at":"2015-01-06T18:13:30Z","updated_at":"2015-01-06T18:13:30Z","body":"HI @pwendell, thanks for taking a look.\r\n\r\nMy reasoning for adding a new env variable for the distribution's class path is twofold:\r\n\r\n- It's not a user option; users are not expected to set it. This is to be set by Spark distributions and never be changed. Of course users can go and modify it just like they can modify other things which are not supposed to be modified, but that's not a supported scenario.\r\n\r\n- It works differently from `spark.{driver,executor}.extraClassPath`. Those options prepend entries to Spark's classpath. The new env variable is appended to the classpath. This makes a huge difference: e.g., if you prepend `/usr/lib/hadoop/lib/*` to Spark's classpath you could break Spark, because that directory contains older versions of some libraries Spark uses.\r\n\r\nAlso, another advantage of not overloading the existing option for this is that you leave that option available to users without them worrying about breaking the distribution's settings.\r\n\r\nAbout the testing classpath, I'll take a second look. Worst case I think I can piggyback on the dist classpath env variable instead of creating a separate one. I just wanted something that did not require me to change every single test (which would potentially be the case if I choose to use the `spark.*.extraClassPath` route).\r\n\r\nI'm currently working on some other changes but I'll try to get back to this by the end of the day."}}