{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9145","labels_url":"https://api.github.com/repos/apache/spark/issues/9145/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9145/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9145/events","html_url":"https://github.com/apache/spark/pull/9145","id":111736958,"number":9145,"title":"[SPARK-11149] [SQL] Improve cache performance for primitive types","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-16T00:57:25Z","updated_at":"2015-10-16T18:47:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9145","html_url":"https://github.com/apache/spark/pull/9145","diff_url":"https://github.com/apache/spark/pull/9145.diff","patch_url":"https://github.com/apache/spark/pull/9145.patch"},"body":"This PR improve the performance by:\r\n\r\n1) Generate an Iterator that take Iterator[CachedBatch] as input, and call accessors (unroll the loop for columns), avoid the expensive Iterator.flatMap.\r\n\r\n2) Use Unsafe.getInt/getLong/getFloat/getDouble instead of ByteBuffer.getInt/getLong/getFloat/getDouble, the later one actually read byte by byte.\r\n\r\n3) Remove the unnecessary copy() in Coalesce(), which is not related to memory cache, found during benchmark.\r\n\r\nThe following benchmark showed that we can speedup the columnar cache of int by 2x.\r\n\r\n```\r\npath = '/opt/tpcds/store_sales/'\r\nint_cols = ['ss_sold_date_sk', 'ss_sold_time_sk', 'ss_item_sk','ss_customer_sk']\r\ndf = sqlContext.read.parquet(path).select(int_cols).cache()\r\ndf.count()\r\n\r\nt = time.time()\r\nprint df.select(\"*\")._jdf.queryExecution().toRdd().count()\r\nprint time.time() - t\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/148804511","html_url":"https://github.com/apache/spark/pull/9145#issuecomment-148804511","issue_url":"https://api.github.com/repos/apache/spark/issues/9145","id":148804511,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T18:47:12Z","updated_at":"2015-10-16T18:47:12Z","body":" Merged build triggered."}}