{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5354","labels_url":"https://api.github.com/repos/apache/spark/issues/5354/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5354/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5354/events","html_url":"https://github.com/apache/spark/pull/5354","id":66209846,"number":5354,"title":"[SPARK-6122][Core] Upgrade tachyon-client version to 0.6.3","user":{"login":"calvinjia","id":2223813,"avatar_url":"https://avatars.githubusercontent.com/u/2223813?v=3","gravatar_id":"","url":"https://api.github.com/users/calvinjia","html_url":"https://github.com/calvinjia","followers_url":"https://api.github.com/users/calvinjia/followers","following_url":"https://api.github.com/users/calvinjia/following{/other_user}","gists_url":"https://api.github.com/users/calvinjia/gists{/gist_id}","starred_url":"https://api.github.com/users/calvinjia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvinjia/subscriptions","organizations_url":"https://api.github.com/users/calvinjia/orgs","repos_url":"https://api.github.com/users/calvinjia/repos","events_url":"https://api.github.com/users/calvinjia/events{/privacy}","received_events_url":"https://api.github.com/users/calvinjia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":47,"created_at":"2015-04-03T19:06:15Z","updated_at":"2015-04-20T18:25:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5354","html_url":"https://github.com/apache/spark/pull/5354","diff_url":"https://github.com/apache/spark/pull/5354.diff","patch_url":"https://github.com/apache/spark/pull/5354.patch"},"body":"This is a reopening of #4867.\r\nA short summary of the issues resolved from the previous PR:\r\n\r\n1. HTTPClient version mismatch: Selenium (used for UI tests) requires version 4.3.x, and Tachyon included 4.2.5 through a transitive dependency of its shaded thrift jar. To address this, Tachyon 0.6.3 will promote the transitive dependencies of the shaded jar so they can be excluded in spark.\r\n\r\n2. Jackson-Mapper-ASL version mismatch: In lower versions of hadoop-client (ie. 1.0.4), version 1.0.1 is included. The parquet library used in spark sql requires version 1.8+. Its unclear to me why upgrading tachyon-client would cause this dependency to break. The solution was to exclude jackson-mapper-asl from hadoop-client. \r\n\r\nIt seems that the dependency management in spark-parent will not work on transitive dependencies, one way to make sure jackson-mapper-asl is included with the correct version is to add it as a top level dependency. The best solution would be to exclude the dependency in the modules which require a higher version, but that did not fix the unit tests. Any suggestions on the best way to solve this would be appreciated!"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/94530456","html_url":"https://github.com/apache/spark/pull/5354#issuecomment-94530456","issue_url":"https://api.github.com/repos/apache/spark/issues/5354","id":94530456,"user":{"login":"calvinjia","id":2223813,"avatar_url":"https://avatars.githubusercontent.com/u/2223813?v=3","gravatar_id":"","url":"https://api.github.com/users/calvinjia","html_url":"https://github.com/calvinjia","followers_url":"https://api.github.com/users/calvinjia/followers","following_url":"https://api.github.com/users/calvinjia/following{/other_user}","gists_url":"https://api.github.com/users/calvinjia/gists{/gist_id}","starred_url":"https://api.github.com/users/calvinjia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvinjia/subscriptions","organizations_url":"https://api.github.com/users/calvinjia/orgs","repos_url":"https://api.github.com/users/calvinjia/repos","events_url":"https://api.github.com/users/calvinjia/events{/privacy}","received_events_url":"https://api.github.com/users/calvinjia/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T18:25:05Z","updated_at":"2015-04-20T18:25:05Z","body":"@srowen \r\nDespite Jackson being managed by dependencyManagement, lower versions make it into managed_libs which cause issues when `mllib` tests run, since they expect the higher version. This is why I exclude the library from hadoop-client which brings in the lower versions.\r\n\r\nFor `httpclient`, `4.2.5` is brought in by `sql/hive` when they reference `libthrift`. This causes selenium tests to break since it expects `4.3.2`.\r\n\r\nFinding the perfect way to reconcile Spark's dependencies as you mentioned before, deserves its own ticket and will require substantial effort. For example, it is not even easy to tell that different versions of `httpclient` are referenced in Spark.\r\n\r\n@ScrapCodes \r\nLet's address cleaning up the Kinesis profile in a separate PR."}}