{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5668","labels_url":"https://api.github.com/repos/apache/spark/issues/5668/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5668/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5668/events","html_url":"https://github.com/apache/spark/pull/5668","id":70477924,"number":5668,"title":"[SPARK-7097][SQL]: Partitioned tables should only consider referred partitions in query during size estimation for checking against autoBroadcastJoinThreshold","user":{"login":"saucam","id":1253893,"avatar_url":"https://avatars.githubusercontent.com/u/1253893?v=3","gravatar_id":"","url":"https://api.github.com/users/saucam","html_url":"https://github.com/saucam","followers_url":"https://api.github.com/users/saucam/followers","following_url":"https://api.github.com/users/saucam/following{/other_user}","gists_url":"https://api.github.com/users/saucam/gists{/gist_id}","starred_url":"https://api.github.com/users/saucam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saucam/subscriptions","organizations_url":"https://api.github.com/users/saucam/orgs","repos_url":"https://api.github.com/users/saucam/repos","events_url":"https://api.github.com/users/saucam/events{/privacy}","received_events_url":"https://api.github.com/users/saucam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":29,"created_at":"2015-04-23T18:16:21Z","updated_at":"2015-07-01T12:35:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5668","html_url":"https://github.com/apache/spark/pull/5668","diff_url":"https://github.com/apache/spark/pull/5668.diff","patch_url":"https://github.com/apache/spark/pull/5668.patch"},"body":"This PR attempts to add support for better size estimation in case of partitioned tables so that only the referred partition's size are taken into consideration when testing against autoBroadCastJoinThreshold and  deciding whether to create a broadcast join or shuffle hash join.\r\n\r\nWe can use the values that get stored in the hive metastore during alter table / insert into partition commands to estimate the size of each of the referred partitions. \r\n\r\nIn most cases, since both alter table query and 'insert into table <tablename> partition <part=val> select * from .....'    store the partition size in the metastore automatically, we expect to get the correct value of partition size. We could use Analyze table query as well in case there is some mismatch. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/117642038","html_url":"https://github.com/apache/spark/pull/5668#issuecomment-117642038","issue_url":"https://api.github.com/repos/apache/spark/issues/5668","id":117642038,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-07-01T12:35:23Z","updated_at":"2015-07-01T12:35:23Z","body":"  [Test build #36258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36258/console) for   PR 5668 at commit [`75fc3e7`](https://github.com/apache/spark/commit/75fc3e7495f314516ca0de5ffd35883fd50323b4).\n * This patch **fails Scala style tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}