{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7216","labels_url":"https://api.github.com/repos/apache/spark/issues/7216/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7216/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7216/events","html_url":"https://github.com/apache/spark/pull/7216","id":92941440,"number":7216,"title":"[SPARK-6910] [SQL] Support for pushing predicates down to metastore for partition pruning","user":{"login":"piaozhexiu","id":179618,"avatar_url":"https://avatars.githubusercontent.com/u/179618?v=3","gravatar_id":"","url":"https://api.github.com/users/piaozhexiu","html_url":"https://github.com/piaozhexiu","followers_url":"https://api.github.com/users/piaozhexiu/followers","following_url":"https://api.github.com/users/piaozhexiu/following{/other_user}","gists_url":"https://api.github.com/users/piaozhexiu/gists{/gist_id}","starred_url":"https://api.github.com/users/piaozhexiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piaozhexiu/subscriptions","organizations_url":"https://api.github.com/users/piaozhexiu/orgs","repos_url":"https://api.github.com/users/piaozhexiu/repos","events_url":"https://api.github.com/users/piaozhexiu/events{/privacy}","received_events_url":"https://api.github.com/users/piaozhexiu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-07-03T21:54:13Z","updated_at":"2015-07-11T08:06:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7216","html_url":"https://github.com/apache/spark/pull/7216","diff_url":"https://github.com/apache/spark/pull/7216.diff","patch_url":"https://github.com/apache/spark/pull/7216.patch"},"body":"This PR supersedes my old one #6921. Since my patch has changed quite a bit, I am opening a new PR to make it easier to review.\r\n\r\nThe changes include-\r\n* Implement `toMetastoreFilter()` function in `HiveShim` that takes `Seq[Expression]` and converts them into a filter string for Hive metastore.\r\n * This functions matches all the `AttributeReference` + `BinaryComparisonOp` + `Integral/StringType` patterns in `Seq[Expression]` and fold them into a string.\r\n* Change `hiveQlPartitions` field in `MetastoreRelation` to `getHiveQlPartitions()` function that takes a filter string parameter.\r\n* Call `getHiveQlPartitions()` in `HiveTableScan` with a filter string.\r\n\r\nBut there are some cases in which predicate pushdown is disabled-\r\n\r\nCase | Predicate pushdown\r\n------- | -----------------------------\r\nHive integral and string types | Yes\r\nHive varchar type | No\r\nHive 0.13 and newer | Yes\r\nHive 0.12 and older | No\r\nconvertMetastoreParquet=false | Yes\r\nconvertMetastoreParquet=true | No\r\n\r\nIn case of `convertMetastoreParquet=true`, predicates are not pushed down because this conversion happens in an `Analyzer` rule (`HiveMetastoreCatalog.ParquetConversions`). At this point, `HiveTableScan` hasn't run, so predicates are not available. But reading the source code, I think it is intentional to convert the entire Hive table w/ all the partitions into `ParquetRelation` because then `ParquetRelation` can be cached and reused for any query against that table. Please correct me if I am wrong.\r\n\r\ncc @marmbrus "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/120593881","html_url":"https://github.com/apache/spark/pull/7216#issuecomment-120593881","issue_url":"https://api.github.com/repos/apache/spark/issues/7216","id":120593881,"user":{"login":"piaozhexiu","id":179618,"avatar_url":"https://avatars.githubusercontent.com/u/179618?v=3","gravatar_id":"","url":"https://api.github.com/users/piaozhexiu","html_url":"https://github.com/piaozhexiu","followers_url":"https://api.github.com/users/piaozhexiu/followers","following_url":"https://api.github.com/users/piaozhexiu/following{/other_user}","gists_url":"https://api.github.com/users/piaozhexiu/gists{/gist_id}","starred_url":"https://api.github.com/users/piaozhexiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piaozhexiu/subscriptions","organizations_url":"https://api.github.com/users/piaozhexiu/orgs","repos_url":"https://api.github.com/users/piaozhexiu/repos","events_url":"https://api.github.com/users/piaozhexiu/events{/privacy}","received_events_url":"https://api.github.com/users/piaozhexiu/received_events","type":"User","site_admin":false},"created_at":"2015-07-11T08:06:02Z","updated_at":"2015-07-11T08:06:02Z","body":"@marmbrus Thank you for reviewing my patch. I think I addressed all your comments. Can you please take a look again?\r\n\r\nBtw, the last jenkins failures seems unrelated to my PR. In fact, jenkins build had succeeded first and then failed in the 2nd run triggered by my minor commit that fixed imports order."}}