{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/presto/issues/2531","labels_url":"https://api.github.com/repos/facebook/presto/issues/2531/labels{/name}","comments_url":"https://api.github.com/repos/facebook/presto/issues/2531/comments","events_url":"https://api.github.com/repos/facebook/presto/issues/2531/events","html_url":"https://github.com/facebook/presto/pull/2531","id":62183736,"number":2531,"title":"Relax partitioning criteria","user":{"login":"martint","id":14387,"avatar_url":"https://avatars.githubusercontent.com/u/14387?v=3","gravatar_id":"","url":"https://api.github.com/users/martint","html_url":"https://github.com/martint","followers_url":"https://api.github.com/users/martint/followers","following_url":"https://api.github.com/users/martint/following{/other_user}","gists_url":"https://api.github.com/users/martint/gists{/gist_id}","starred_url":"https://api.github.com/users/martint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martint/subscriptions","organizations_url":"https://api.github.com/users/martint/orgs","repos_url":"https://api.github.com/users/martint/repos","events_url":"https://api.github.com/users/martint/events{/privacy}","received_events_url":"https://api.github.com/users/martint/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-16T19:21:35Z","updated_at":"2015-04-02T02:11:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/facebook/presto/pulls/2531","html_url":"https://github.com/facebook/presto/pull/2531","diff_url":"https://github.com/facebook/presto/pull/2531.diff","patch_url":"https://github.com/facebook/presto/pull/2531.patch"},"body":"A plan that is partitioned on a set of columns C satisfies partitioning requirements on any superset of C.\r\n\r\nThis change removes unnecessary shuffles for certain queries. E.g.,\r\n\r\n```sql\r\n   SELECT orderdate, c\r\n   FROM (\r\n      SELECT orderdate, count(*) c FROM tpch.sf1.orders\r\n      GROUP BY orderdate)\r\n   GROUP BY orderdate, c;\r\n```\r\nBefore:\r\n\r\n```\r\n Fragment 3 [SINGLE]\r\n     Output layout: [orderdate, count]\r\n     - Output[orderdate, c] => [orderdate:date, count:bigint]\r\n             c := count\r\n         - RemoteSource[2] => [orderdate:date, count:bigint]\r\n\r\n Fragment 2 [FIXED]\r\n     Output layout: [orderdate, count]\r\n     - Aggregate(FINAL)[orderdate, count] => [orderdate:date, count:bigint]\r\n         - RemoteSource[1] => [orderdate:date, count:bigint]\r\n\r\n Fragment 1 [FIXED]\r\n     Output layout: [orderdate, count]\r\n     Output partitioning: [orderdate, count]\r\n     - Aggregate(PARTIAL)[orderdate, count] => [orderdate:date, count:bigint]\r\n         - Aggregate(FINAL)[orderdate] => [orderdate:date, count:bigint]\r\n                 count := \"count\"(\"count_18\")\r\n             - RemoteSource[0] => [orderdate:date, count_18:bigint]\r\n\r\n Fragment 0 [SOURCE]\r\n     Output layout: [orderdate, count_18]\r\n     Output partitioning: [orderdate]\r\n     - Aggregate(PARTIAL)[orderdate] => [orderdate:date, count_18:bigint]\r\n             count_18 := \"count\"(*)\r\n         - TableScan[tpch:tpch:orders:sf1.0, original constraint=true] => [orderdate:date]\r\n                 orderdate := tpch:tpch:orderdate:4\r\n```\r\n\r\nAfter:\r\n\r\n```\r\n Fragment 2 [SINGLE]\r\n     Output layout: [orderdate, count]\r\n     - Output[orderdate, c] => [orderdate:date, count:bigint]\r\n             c := count\r\n         - RemoteSource[1] => [orderdate:date, count:bigint]\r\n\r\n Fragment 1 [FIXED]\r\n     Output layout: [orderdate, count]\r\n     - Aggregate[orderdate, count] => [orderdate:date, count:bigint]\r\n         - Aggregate(FINAL)[orderdate] => [orderdate:date, count:bigint]\r\n                 count := \"count\"(\"count_18\")\r\n             - RemoteSource[0] => [orderdate:date, count_18:bigint]\r\n\r\n Fragment 0 [SOURCE]\r\n     Output layout: [orderdate, count_18]\r\n     Output partitioning: [orderdate]\r\n     - Aggregate(PARTIAL)[orderdate] => [orderdate:date, count_18:bigint]\r\n             count_18 := \"count\"(*)\r\n         - TableScan[tpch:tpch:orders:sf1.0, original constraint=true] => [orderdate:date]\r\n                 orderdate := tpch:tpch:orderdate:4\r\n```"},"comment":{"url":"https://api.github.com/repos/facebook/presto/issues/comments/88697057","html_url":"https://github.com/facebook/presto/pull/2531#issuecomment-88697057","issue_url":"https://api.github.com/repos/facebook/presto/issues/2531","id":88697057,"user":{"login":"erichwang","id":737851,"avatar_url":"https://avatars.githubusercontent.com/u/737851?v=3","gravatar_id":"","url":"https://api.github.com/users/erichwang","html_url":"https://github.com/erichwang","followers_url":"https://api.github.com/users/erichwang/followers","following_url":"https://api.github.com/users/erichwang/following{/other_user}","gists_url":"https://api.github.com/users/erichwang/gists{/gist_id}","starred_url":"https://api.github.com/users/erichwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erichwang/subscriptions","organizations_url":"https://api.github.com/users/erichwang/orgs","repos_url":"https://api.github.com/users/erichwang/repos","events_url":"https://api.github.com/users/erichwang/events{/privacy}","received_events_url":"https://api.github.com/users/erichwang/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T02:11:51Z","updated_at":"2015-04-02T02:11:51Z","body":"I'm not sure we can make this optimization in the general case. This is less apparent in aggregations, but in joins this can be a larger problem. For example if the source is partitioned on column A for some reason and then you want to do a join on (A, B), the cardinality of A by itself may have really bad skew and this could result in really bad join performance if it is not repartitioned. This is probably only a decision that can be made with cost and estimates."}}