{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6921","labels_url":"https://api.github.com/repos/apache/spark/issues/6921/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6921/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6921/events","html_url":"https://github.com/apache/spark/pull/6921","id":89797549,"number":6921,"title":"[SPARK-6910] [SQL] [WIP] Support for pushing predicates down to metastore for partition pruning","user":{"login":"piaozhexiu","id":179618,"avatar_url":"https://avatars.githubusercontent.com/u/179618?v=3","gravatar_id":"","url":"https://api.github.com/users/piaozhexiu","html_url":"https://github.com/piaozhexiu","followers_url":"https://api.github.com/users/piaozhexiu/followers","following_url":"https://api.github.com/users/piaozhexiu/following{/other_user}","gists_url":"https://api.github.com/users/piaozhexiu/gists{/gist_id}","starred_url":"https://api.github.com/users/piaozhexiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piaozhexiu/subscriptions","organizations_url":"https://api.github.com/users/piaozhexiu/orgs","repos_url":"https://api.github.com/users/piaozhexiu/repos","events_url":"https://api.github.com/users/piaozhexiu/events{/privacy}","received_events_url":"https://api.github.com/users/piaozhexiu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-06-20T18:01:06Z","updated_at":"2015-06-22T03:17:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6921","html_url":"https://github.com/apache/spark/pull/6921","diff_url":"https://github.com/apache/spark/pull/6921.diff","patch_url":"https://github.com/apache/spark/pull/6921.patch"},"body":"@marmbrus per our email conversation, I am sending a PR that implements the idea that I outlined in the jira FYI.\r\n* `Analyzer` captures predicates if a `unresolved relation` is followed by a `filter expression` and pushes them down into `HiveMetastoreCatalog`.\r\n* `HiveMetastoreCatalog` extracts partition predicates whose types are string and integral and constructs a string representation for them.\r\n* Hive client invokes `getPartitionsByFilter(Table tbl, String filter)` with the filter string.\r\n\r\nIn addition, I understand better the limitations of Hive `getPartitionsByFilter(Table tbl, String filter)` function now.\r\n* Good news: it works with integral types as well as string.\r\n* Bad news: it only works with equals operators.\r\n\r\nThe following is where the *\"filtering is supported only on partition keys of type string\"* error from in Hive-\r\n```java\r\n// metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java\r\npublic String getFilterPushdownParam(Table table, int partColIndex) throws MetaException {\r\n  boolean isIntegralSupported = doesOperatorSupportIntegral(operator);\r\n  String colType = table.getPartitionKeys().get(partColIndex).getType();\r\n  // Can only support partitions whose types are string, or maybe integers\r\n  if (!colType.equals(serdeConstants.STRING_TYPE_NAME)\r\n      && (!isIntegralSupported || !isIntegralType(colType))) {\r\n    throw new MetaException(\"Filtering is supported only on partition keys of type \" +\r\n        \"string\" + (isIntegralSupported ? \", or integral types\" : \"\"));\r\n  }\r\n  boolean isStringValue = value instanceof String;\r\n  if (!isStringValue && (!isIntegralSupported || !(value instanceof Long))) {\r\n    throw new MetaException(\"Filtering is supported only on partition keys of type \" +\r\n        \"string\" + (isIntegralSupported ? \", or integral types\" : \"\"));\r\n  }\r\n  return isStringValue ? (String) value : Long.toString((Long) value);\r\n}\r\n```\r\nAs can be seen, one of following conditions has to be met-\r\n* Partition key is string type; or\r\n* Partition key is integral type, and operator supports integral types. (i.e `doesOperatorSupportIntegral()` returns true.) \r\n\r\nBut looking at the `doesOperatorSupportIntegral()` function, only equals and  not-equals operators support integral types-\r\n```java\r\n// metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java\r\nprivate static boolean doesOperatorSupportIntegral(Operator operator) {\r\n  // TODO: for SQL-based filtering, this could be amended if we added casts.\r\n  return (operator == Operator.EQUALS)\r\n      || (operator == Operator.NOTEQUALS)\r\n      || (operator == Operator.NOTEQUALS2);\r\n}\r\n```\r\nThis is not good because `<`, `>`, `<=`, and `>=` which are very common in practice are not supported.\r\n\r\nIn this PR, I restricted predicate pushdown to `=`. With that restriction, all the unit tests pass now."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/113990502","html_url":"https://github.com/apache/spark/pull/6921#issuecomment-113990502","issue_url":"https://api.github.com/repos/apache/spark/issues/6921","id":113990502,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-06-22T03:17:18Z","updated_at":"2015-06-22T03:17:18Z","body":"Merged build started."}}