{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5668","labels_url":"https://api.github.com/repos/apache/spark/issues/5668/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5668/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5668/events","html_url":"https://github.com/apache/spark/pull/5668","id":70477924,"number":5668,"title":"[SPARK-7097][SQL]: Partitioned tables should only consider referred partitions in query during size estimation for checking against autoBroadcastJoinThreshold","user":{"login":"saucam","id":1253893,"avatar_url":"https://avatars.githubusercontent.com/u/1253893?v=3","gravatar_id":"","url":"https://api.github.com/users/saucam","html_url":"https://github.com/saucam","followers_url":"https://api.github.com/users/saucam/followers","following_url":"https://api.github.com/users/saucam/following{/other_user}","gists_url":"https://api.github.com/users/saucam/gists{/gist_id}","starred_url":"https://api.github.com/users/saucam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saucam/subscriptions","organizations_url":"https://api.github.com/users/saucam/orgs","repos_url":"https://api.github.com/users/saucam/repos","events_url":"https://api.github.com/users/saucam/events{/privacy}","received_events_url":"https://api.github.com/users/saucam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-04-23T18:16:21Z","updated_at":"2015-04-23T18:31:31Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5668","html_url":"https://github.com/apache/spark/pull/5668","diff_url":"https://github.com/apache/spark/pull/5668.diff","patch_url":"https://github.com/apache/spark/pull/5668.patch"},"body":"This PR attempts to add support for better size estimation in case of partitioned tables so that only the referred partition's size are taken into consideration when testing against autoBroadCastJoinThreshold and  deciding whether to create a broadcast join or shuffle hash join.\r\n\r\nWe can use the values that get stored in the hive metastore during alter table / insert into partition commands to estimate the size of each of the referred partitions. In case the size returned is 0, we list the files in the partition to get the size, store it in the metastore for reuse and continue with the analysis.\r\n\r\nIn most cases, since both alter table query and 'insert into table <tablename> partition <part=val> select * from .....'  store the partition size in the metastore automatically, we would not require to compute the size by listing files in the directory. We could use Analyze table query as well in case re computation is required. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/95678594","html_url":"https://github.com/apache/spark/pull/5668#issuecomment-95678594","issue_url":"https://api.github.com/repos/apache/spark/issues/5668","id":95678594,"user":{"login":"saucam","id":1253893,"avatar_url":"https://avatars.githubusercontent.com/u/1253893?v=3","gravatar_id":"","url":"https://api.github.com/users/saucam","html_url":"https://github.com/saucam","followers_url":"https://api.github.com/users/saucam/followers","following_url":"https://api.github.com/users/saucam/following{/other_user}","gists_url":"https://api.github.com/users/saucam/gists{/gist_id}","starred_url":"https://api.github.com/users/saucam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saucam/subscriptions","organizations_url":"https://api.github.com/users/saucam/orgs","repos_url":"https://api.github.com/users/saucam/repos","events_url":"https://api.github.com/users/saucam/events{/privacy}","received_events_url":"https://api.github.com/users/saucam/received_events","type":"User","site_admin":false},"created_at":"2015-04-23T18:31:30Z","updated_at":"2015-04-23T18:31:30Z","body":"retest please"}}