{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9981","labels_url":"https://api.github.com/repos/apache/spark/issues/9981/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9981/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9981/events","html_url":"https://github.com/apache/spark/pull/9981","id":118936443,"number":9981,"title":"[SPARK-11619][SQL] cannot use UDTF in DataFrame.selectExpr","user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-11-25T22:43:22Z","updated_at":"2015-12-01T06:59:14Z","closed_at":"2015-11-30T03:55:41Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9981","html_url":"https://github.com/apache/spark/pull/9981","diff_url":"https://github.com/apache/spark/pull/9981.diff","patch_url":"https://github.com/apache/spark/pull/9981.patch"},"body":"Description of the problem from @cloud-fan \r\n\r\nActually this line: https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L689\r\nWhen we use `selectExpr`, we pass in `UnresolvedFunction` to `DataFrame.select` and fall in the last case. A workaround is to do special handling for UDTF like we did for `explode`(and `json_tuple` in 1.6), wrap it with `MultiAlias`.\r\nAnother workaround is using `expr`, for example, `df.select(expr(\"explode(a)\").as(Nil))`, I think `selectExpr` is no longer needed after we have the `expr` function...."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/160877804","html_url":"https://github.com/apache/spark/pull/9981#issuecomment-160877804","issue_url":"https://api.github.com/repos/apache/spark/issues/9981","id":160877804,"user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T06:59:13Z","updated_at":"2015-12-01T06:59:13Z","body":"@cloud-fan Hi Wenchen, I was thinking about this and want to run an idea by you. Is it ok if we add the logic to inject the MultiAlias in our analyzer. As an experiment, i put the following code in ResolveGenerate() and it seems to work and also i am running the test suites to make sure. \r\n\r\nval newProj : Seq[NamedExpression] = projectList.map { expr =>\r\n          expr match  {\r\n            case a @ Alias(g: Generator, name) if g.resolved &&\r\n              g.elementTypes.size > 1 && name.equals(a.child.prettyString) =>\r\n              MultiAlias(g, Nil)\r\n            case e @ _ => e\r\n          }\r\n        }\r\n\r\nOne thing is name.equals(a.child.prettyString) does not really make sure if its an user specified alias or a system generated one. We can flag an alias to differentiate if required. Just wanted to know what you thought about this approach. Thanks in advance."}}