{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7564","labels_url":"https://api.github.com/repos/apache/spark/issues/7564/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7564/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7564/events","html_url":"https://github.com/apache/spark/pull/7564","id":96244983,"number":7564,"title":"[SPARK-8850] [SQL] [WIP] Enable Unsafe mode by default","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":62,"created_at":"2015-07-21T07:00:17Z","updated_at":"2015-07-27T19:37:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7564","html_url":"https://github.com/apache/spark/pull/7564","diff_url":"https://github.com/apache/spark/pull/7564.diff","patch_url":"https://github.com/apache/spark/pull/7564.patch"},"body":"The purpose of this WIP pull request is to figure out which changes will be necessary to enable Unsafe mode by default in Spark SQL.\r\n\r\n**List of current blockers**:\r\n\r\n\r\n- [ ] Make some default buffer sizes configurable so that HiveCompatibilitySuite can run properly.\r\n- After having lowered the buffer sizes to 4MB so that most of HiveCompatibilitySuite runs:\r\n  - [x] Wrong answer in `join_1to1` (fixed by #7680)\r\n  - [x] Wrong answer in `join_nulls` (fixed by #7680)\r\n  - [x] Managed memory OOM / leak in `lateral_view`\r\n  - [ ] Seems to hang indefinitely in `partcols1`.  This might be a deadlock in script transformation or a bug in error-handling code?\r\n\r\n\r\n**List of fixed blockers**:\r\n\r\n- [x] Memory leak on grouped aggregation of empty input (fixed by #7560 to fix this)\r\n- [x] Update planner to also check whether codegen is enabled before planning unsafe operators.\r\n- [x] Investigate failing HiveThriftBinaryServerSuite test.  This turns out to be caused by a ClassCastException that occurs when Exchange tries to apply an interpreted RowOrdering to an UnsafeRow when range partitioning an RDD.  This could be fixed by #7408, but a shorter-term fix is to just skip the Unsafe exchange path when RangePartitioner is used.\r\n- [x] Memory leak exceptions masking exceptions that actually caused tasks to fail (will be fixed by #7603).\r\n- [x]  ~~https://issues.apache.org/jira/browse/SPARK-9162, to implement code generation for ScalaUDF.  This is necessary for `UDFSuite` to pass.  For now, I've just ignored this test in order to try to find other problems while we wait for a fix.~~ This is no longer necessary as of #7682.\r\n- [x] Memory leaks from Limit after UnsafeExternalSort cause the memory leak detector to fail tests. This is a huge problem in the HiveCompatibilitySuite (fixed by f4ac642a4e5b2a7931c5e04e086bb10e263b1db6).\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/125320416","html_url":"https://github.com/apache/spark/pull/7564#issuecomment-125320416","issue_url":"https://api.github.com/repos/apache/spark/issues/7564","id":125320416,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-07-27T19:37:01Z","updated_at":"2015-07-27T19:37:01Z","body":"Merged build finished. Test FAILed."}}