{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5203","labels_url":"https://api.github.com/repos/apache/spark/issues/5203/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5203/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5203/events","html_url":"https://github.com/apache/spark/pull/5203","id":64469808,"number":5203,"title":"[SPARK-6550][SQL] Add PreAnalyzer to keep logical plan consistent across DataFrame","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-26T08:35:08Z","updated_at":"2015-03-26T08:54:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5203","html_url":"https://github.com/apache/spark/pull/5203","diff_url":"https://github.com/apache/spark/pull/5203.diff","patch_url":"https://github.com/apache/spark/pull/5203.patch"},"body":"## Problems\r\n\r\nIn some cases, the expressions in a logical plan will be modified to new ones during analysis, e.g. the handling for self-join cases. If some expressions are resolved based on the analyzed plan, they are referring to changed expression ids, not original ids.\r\n\r\nBut the transformation of DataFrame will use logical plan to construct new DataFrame, e.g. `groupBy` and aggregation. So in such cases, the expressions in these DataFrames will be inconsistent.\r\n\r\nThe problems are specified as following:\r\n\r\n* Expression ids in logical plan are possibly inconsistent if expression ids are changed during analysis and some expressions are resolved after that\r\n\r\nWhen we try to run the following codes:\r\n\r\n    val df = Seq(1,2,3).map(i => (i, i.toString)).toDF(\"int\", \"str\")\r\n    val df2 = df.as('x).join(df.as('y), $\"x.str\" === $\"y.str\").groupBy(\"y.str\").min(\"y.int\")\r\n\r\n\r\nBecause `groupBy` and `min` will perform resolving based on the analyzed logical plan, their expression ids refer to analyzed plan, instead of logical plan.\r\n\r\nSo the logical plan of df2 looks like:\r\n\r\n    'Aggregate [str#5], [str#5,MIN(int#4) AS MIN(int)#6]\r\n     'Join Inner, Some(('x.str = 'y.str))\r\n      Subquery x\r\n       Project [_1#0 AS int#2,_2#1 AS str#3]\r\n        LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3]]\r\n      Subquery y\r\n       Project [_1#0 AS int#2,_2#1 AS str#3]\r\n        LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3]]\r\n\r\n\r\nAs you see, the expression ids in `Aggregate` are different to the expression ids in `Subquery y`. This is the first problem.\r\n\r\n* The `df2` can't be performed\r\n\r\nThe showing logical plan of `df2` can't be performed. Because the expression ids of `Subquery y` will be modified for self-join handling during analysis, the analyzed plan of `df2` becomes:\r\n\r\n\r\n    Aggregate [str#5], [str#5,MIN(int#4) AS MIN(int)#6]\r\n     Join Inner, Some((str#3 = str#8))\r\n      Subquery x\r\n       Project [_1#0 AS int#2,_2#1 AS str#3]\r\n        LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3]]\r\n      Subquery y\r\n       Project [_1#0 AS int#7,_2#1 AS str#8]\r\n        LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3]]\r\n\r\n\r\nThe expressions referred in `Aggregate` are not matching to these in `Subquery y`. This is the second problem.\r\n\r\n## Proposed solution\r\n\r\nWe try to add a `PreAnalyzer`. When a logical plan `rawPlan` is given to SQLContext, it uses PreAnalyzer to modify the logical plan before assigning to `QueryExecution.logical`. Then later operations will based on the pre-analyzed logical plan, instead of the original `rawPlan`.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/86408687","html_url":"https://github.com/apache/spark/pull/5203#issuecomment-86408687","issue_url":"https://api.github.com/repos/apache/spark/issues/5203","id":86408687,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-03-26T08:54:41Z","updated_at":"2015-03-26T08:54:41Z","body":"  [Test build #29219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29219/consoleFull) for   PR 5203 at commit [`77ba3a6`](https://github.com/apache/spark/commit/77ba3a6dba84928935cf18c2e4d9b40a03ac404f).\n * This patch **fails Spark unit tests**.\n\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `class PreAnalyzer(caseSensitive: Boolean = true,`\n  * `  protected[sql] class QueryExecution(val rawPlan: LogicalPlan) `\n  * `  protected[sql] class QueryExecution(rawPlan: LogicalPlan)`\n"}}