{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9575","labels_url":"https://api.github.com/repos/apache/spark/issues/9575/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9575/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9575/events","html_url":"https://github.com/apache/spark/pull/9575","id":115980103,"number":9575,"title":"[SPARK-7841][BUILD] Stop using retrieveManaged to retrieve dependencies in SBT","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-09T22:17:38Z","updated_at":"2015-11-10T02:59:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9575","html_url":"https://github.com/apache/spark/pull/9575","diff_url":"https://github.com/apache/spark/pull/9575.diff","patch_url":"https://github.com/apache/spark/pull/9575.patch"},"body":"This patch modifies Spark's SBT build so that it no longer uses `retrieveManaged` / `lib_managed` to store its dependencies. The motivations for this change are nicely described on the JIRA ticket ([SPARK-7841](https://issues.apache.org/jira/browse/SPARK-7841)); my personal interest in doing this stems from the fact that `lib_managed` has caused me some pain while debugging dependency issues in another PR of mine.\r\n\r\nRemoving our use of `lib_managed` would be trivial except for one snag: the Datanucleus JARs, required by Spark SQL's Hive integration, cannot be included in assembly JARs due to problems with merging OSGI `plugin.xml` files. As a result, several places in the packaging and deployment pipeline assume that these Datanucleus JARs are copied to `lib_managed/jars`. In the interest of maintaining compatibility, I have chosen to retain the `lib_managed/jars` directory _only_ for these Datanucleus JARs and have added custom code to `SparkBuild.scala` to automatically copy those JARs to that folder as part of the `assembly` task.\r\n\r\n`dev/mima` also depended on `lib_managed` in a hacky way in order to set classpaths when generating MiMa excludes; I've updated this to obtain the classpaths directly from SBT instead.\r\n\r\n/cc @dragos @marmbrus @pwendell @srowen "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/155268339","html_url":"https://github.com/apache/spark/pull/9575#issuecomment-155268339","issue_url":"https://api.github.com/repos/apache/spark/issues/9575","id":155268339,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T02:59:51Z","updated_at":"2015-11-10T02:59:51Z","body":"**[Test build #45444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45444/consoleFull)** for PR 9575 at commit [`828c3b4`](https://github.com/apache/spark/commit/828c3b4207c2a1d01bf443f1e41a041ed6a982f1).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\\n  * `class MasterWebUI(`\\n  * `public class JavaAFTSurvivalRegressionExample `\\n"}}