{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8295","labels_url":"https://api.github.com/repos/apache/spark/issues/8295/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8295/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8295/events","html_url":"https://github.com/apache/spark/pull/8295","id":101786628,"number":8295,"title":"[SPARK-10093][SPARK-10096][SQL] Avoid transformation on executors & fix UDFS on complex types","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-08-19T01:39:05Z","updated_at":"2015-08-19T04:04:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8295","html_url":"https://github.com/apache/spark/pull/8295","diff_url":"https://github.com/apache/spark/pull/8295.diff","patch_url":"https://github.com/apache/spark/pull/8295.patch"},"body":"This is kind of a weird case, but given a sufficiently complex query plan (in this case a TungstenProject with an Exchange underneath), we could have NPEs on the executors due to the time when we were calling transformAllExpressions\r\n\r\nIn general we should ensure that all transformations occur on the driver and not on the executors. Some reasons for avoid executor side transformations include:\r\n\r\n* (this case) Some operator constructors require state such as access to the Spark/SQL conf so doing a makeCopy on the executor can fail.\r\n* (unrelated reason for avoid executor transformations) ExprIds are calculated using an atomic integer, so you can violate their uniqueness constraint by constructing them anywhere other than the driver.\r\n\r\nThis subsumes #8285.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/132439686","html_url":"https://github.com/apache/spark/pull/8295#issuecomment-132439686","issue_url":"https://api.github.com/repos/apache/spark/issues/8295","id":132439686,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-19T04:04:52Z","updated_at":"2015-08-19T04:04:52Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41192/\nTest PASSed."}}