{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8971","labels_url":"https://api.github.com/repos/apache/spark/issues/8971/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8971/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8971/events","html_url":"https://github.com/apache/spark/pull/8971","id":109594713,"number":8971,"title":"[SPARK-10917] [SQL] improve performance of complex type in columnar cache","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-10-03T00:54:43Z","updated_at":"2015-10-07T22:31:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8971","html_url":"https://github.com/apache/spark/pull/8971","diff_url":"https://github.com/apache/spark/pull/8971.diff","patch_url":"https://github.com/apache/spark/pull/8971.patch"},"body":"This PR improve the performance of complex types in columnar cache by using UnsafeProjection instead of KryoSerializer. \r\n\r\nA simple benchmark show that this PR could improve the performance of scanning a cached table with complex columns by 15x (comparing to Spark 1.5).\r\n\r\nHere is the code used to benchmark:\r\n\r\n```\r\ndf = sc.range(1<<23).map(lambda i: Row(a=Row(b=i, c=str(i)), d=range(10), e=dict(zip(range(10), [str(i) for i in range(10)])))).toDF()\r\ndf.write.parquet(\"table\")\r\n```\r\n```\r\ndf = sqlContext.read.parquet(\"table\")\r\ndf.cache()\r\ndf.count()\r\nt = time.time()\r\nprint df.select(\"*\")._jdf.queryExecution().toRdd().count()\r\nprint time.time() - t\r\n```  "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/146351398","html_url":"https://github.com/apache/spark/pull/8971#issuecomment-146351398","issue_url":"https://api.github.com/repos/apache/spark/issues/8971","id":146351398,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-10-07T22:31:10Z","updated_at":"2015-10-07T22:31:10Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43341/\nTest PASSed."}}