{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7920","labels_url":"https://api.github.com/repos/apache/spark/issues/7920/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7920/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7920/events","html_url":"https://github.com/apache/spark/pull/7920","id":98879437,"number":7920,"title":"[SPARK-9141][SQL] Remove project collapsing from DataFrame API","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-04T03:18:16Z","updated_at":"2015-08-04T03:27:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7920","html_url":"https://github.com/apache/spark/pull/7920","diff_url":"https://github.com/apache/spark/pull/7920.diff","patch_url":"https://github.com/apache/spark/pull/7920.patch"},"body":"Currently we collapse successive projections that are added by `withColumn`.  However, this optimization violates the constraint that adding nodes to a plan will never change its analyzed form and thus breaks caching.  Instead of doing early optimization, in this PR I just fix some low-hanging slowness in the analyzer.  In particular, I add a mechanism for skipping already analyzed subplans.  Since trees are generally immutable after construction, its safe to annotate a plan as already analyzed as any transformation will create a new tree with this bit no longer set.  While this slows down slightly from the optimized form it is still pretty fast for experiments with up to 200 columns.\r\n\r\n```\r\nOriginal Code\r\n[info] 3430ms\r\n[info] 2205ms\r\n[info] 1973ms\r\n[info] 1982ms\r\n[info] 1916ms\r\n\r\nWithout Project Collapsing in DataFrame\r\n[info] 44610ms\r\n[info] 45977ms\r\n[info] 46423ms\r\n[info] 46306ms\r\n[info] 54723ms\r\n\r\nWith analyzer optimizations\r\n[info] 6394ms\r\n[info] 4630ms\r\n[info] 4388ms\r\n[info] 4093ms\r\n[info] 4113ms\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127465074","html_url":"https://github.com/apache/spark/pull/7920#issuecomment-127465074","issue_url":"https://api.github.com/repos/apache/spark/issues/7920","id":127465074,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T03:27:50Z","updated_at":"2015-08-04T03:27:50Z","body":" Merged build triggered."}}