{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10451","labels_url":"https://api.github.com/repos/apache/spark/issues/10451/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10451/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10451/events","html_url":"https://github.com/apache/spark/pull/10451","id":123702709,"number":10451,"title":"[SPARK-12503] [SQL] Pushing Limit Through Union All","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-12-23T18:15:39Z","updated_at":"2015-12-29T23:33:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10451","html_url":"https://github.com/apache/spark/pull/10451","diff_url":"https://github.com/apache/spark/pull/10451.diff","patch_url":"https://github.com/apache/spark/pull/10451.patch"},"body":"\"Rule that applies to a Limit on top of a Union. The original Limit won't go away after applying this rule, but additional Limit nodes will be created on top of each child of Union, so that these children produce less rows and Limit can be further optimized for children Relations.\"\r\n\r\n– from https://issues.apache.org/jira/browse/CALCITE-832\r\n\r\nAlso, the same topic in Hive: https://issues.apache.org/jira/browse/HIVE-11775. This has been merged to HIVE. \r\n\r\nThis PR is for performance improvement. The idea is like predicate pushdown. It can reduce the number of rows processed by `Union All`. \r\n\r\nAfter the improvement, we can see the changes in the optimized plan:\r\n```\r\n== Analyzed Logical Plan ==\r\ni: int\r\nLimit 1\r\n+- Union\r\n   :- Project [_1#0 AS i#1]\r\n   :  +- LocalRelation [_1#0], [[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20]]\r\n   +- Project [_1#2 AS i#3]\r\n      +- LocalRelation [_1#2], [[1],[2],[3],[4],[5],[6],[7],[8],[9],[10]]\r\n\r\n== Optimized Logical Plan ==\r\nLimit 1\r\n+- Union\r\n   :- Limit 1 <---- extra Limit\r\n   :  +- LocalRelation [i#1], [[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20]]\r\n   +- Limit 1 <---- extra Limit\r\n      +- LocalRelation [i#3], [[1],[2],[3],[4],[5],[6],[7],[8],[9],[10]]\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/167899814","html_url":"https://github.com/apache/spark/pull/10451#issuecomment-167899814","issue_url":"https://api.github.com/repos/apache/spark/issues/10451","id":167899814,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T23:33:47Z","updated_at":"2015-12-29T23:33:47Z","body":"**[Test build #48431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48431/consoleFull)** for PR 10451 at commit [`2823a57`](https://github.com/apache/spark/commit/2823a57b14705f961e53f53c9341f42752889474)."}}