{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9050","labels_url":"https://api.github.com/repos/apache/spark/issues/9050/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9050/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9050/events","html_url":"https://github.com/apache/spark/pull/9050","id":110736927,"number":9050,"title":"[SPARK-11009] [SQL] fix wrong result of Window function in cluster mode","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-10-09T21:13:26Z","updated_at":"2015-10-13T06:42:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9050","html_url":"https://github.com/apache/spark/pull/9050","diff_url":"https://github.com/apache/spark/pull/9050.diff","patch_url":"https://github.com/apache/spark/pull/9050.patch"},"body":"Currently, All windows function could generate wrong result in cluster sometimes.\r\n\r\nThe root cause is that AttributeReference is called in executor, then id of it may not be unique than others created in driver.\r\n\r\nHere is the script that could reproduce the problem (run in local cluster):\r\n```\r\nfrom pyspark import SparkContext, HiveContext\r\nfrom pyspark.sql.window import Window\r\nfrom pyspark.sql.functions import rowNumber\r\n\r\nsqlContext = HiveContext(SparkContext())\r\nsqlContext.setConf(\"spark.sql.shuffle.partitions\", \"3\")\r\ndf =  sqlContext.range(1<<20)\r\ndf2 = df.select((df.id % 1000).alias(\"A\"), (df.id / 1000).alias('B'))\r\nws = Window.partitionBy(df2.A).orderBy(df2.B)\r\ndf3 = df2.select(\"client\", \"date\", rowNumber().over(ws).alias(\"rn\")).filter(\"rn < 0\")\r\nassert df3.count() == 0\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/147620505","html_url":"https://github.com/apache/spark/pull/9050#issuecomment-147620505","issue_url":"https://api.github.com/repos/apache/spark/issues/9050","id":147620505,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-10-13T06:42:21Z","updated_at":"2015-10-13T06:42:21Z","body":"Merged build started."}}