{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10128","labels_url":"https://api.github.com/repos/apache/spark/issues/10128/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10128/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10128/events","html_url":"https://github.com/apache/spark/pull/10128","id":120137038,"number":10128,"title":"[SPARK-12125][SQL] pull out nondeterministic expressions from Join","user":{"login":"zhonghaihua","id":4409429,"avatar_url":"https://avatars.githubusercontent.com/u/4409429?v=3","gravatar_id":"","url":"https://api.github.com/users/zhonghaihua","html_url":"https://github.com/zhonghaihua","followers_url":"https://api.github.com/users/zhonghaihua/followers","following_url":"https://api.github.com/users/zhonghaihua/following{/other_user}","gists_url":"https://api.github.com/users/zhonghaihua/gists{/gist_id}","starred_url":"https://api.github.com/users/zhonghaihua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhonghaihua/subscriptions","organizations_url":"https://api.github.com/users/zhonghaihua/orgs","repos_url":"https://api.github.com/users/zhonghaihua/repos","events_url":"https://api.github.com/users/zhonghaihua/events{/privacy}","received_events_url":"https://api.github.com/users/zhonghaihua/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-12-03T10:20:59Z","updated_at":"2015-12-03T15:27:05Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10128","html_url":"https://github.com/apache/spark/pull/10128","diff_url":"https://github.com/apache/spark/pull/10128.diff","patch_url":"https://github.com/apache/spark/pull/10128.patch"},"body":"Currently,`nondeterministic expressions` are only allowed in `Project` or `Filter`,And only when we use nondeterministic expressions in `UnaryNode` can be pulled out.\r\n\r\nBut,Sometime in many case,we will use nondeterministic expressions to process `join keys` avoiding data skew.for example:\r\n```\r\nselect * \r\nfrom tableA a \r\njoin \r\n(select * from tableB) b \r\non upper((case when (a.brand_code is null or a.brand_code = '' ) then cast( (-rand() * 10000000 ) as string ) else a.brand_code end ))  = b.brand_code\r\n\r\n```\r\n\r\nThis PR introduce a mechanism to pull out nondeterministic expressions from `Join`,so we can use nondeterministic expression in `Join` appropriately."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/161675474","html_url":"https://github.com/apache/spark/pull/10128#issuecomment-161675474","issue_url":"https://api.github.com/repos/apache/spark/issues/10128","id":161675474,"user":{"login":"chenghao-intel","id":2367031,"avatar_url":"https://avatars.githubusercontent.com/u/2367031?v=3","gravatar_id":"","url":"https://api.github.com/users/chenghao-intel","html_url":"https://github.com/chenghao-intel","followers_url":"https://api.github.com/users/chenghao-intel/followers","following_url":"https://api.github.com/users/chenghao-intel/following{/other_user}","gists_url":"https://api.github.com/users/chenghao-intel/gists{/gist_id}","starred_url":"https://api.github.com/users/chenghao-intel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenghao-intel/subscriptions","organizations_url":"https://api.github.com/users/chenghao-intel/orgs","repos_url":"https://api.github.com/users/chenghao-intel/repos","events_url":"https://api.github.com/users/chenghao-intel/events{/privacy}","received_events_url":"https://api.github.com/users/chenghao-intel/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T15:27:05Z","updated_at":"2015-12-03T15:27:05Z","body":"I can understand the motivation of this change, we do have workaround for relieving the data skew, but we probably don't want to change the existing SQL queries based on legacy system (like Hive)."}}