{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8805","labels_url":"https://api.github.com/repos/apache/spark/issues/8805/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8805/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8805/events","html_url":"https://github.com/apache/spark/pull/8805","id":107105966,"number":8805,"title":"[SPARK-10677] [SQL] [WIP] UnsafeExternalSorter atomic acquire release","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-18T00:17:06Z","updated_at":"2015-09-18T18:52:56Z","closed_at":"2015-09-18T18:52:56Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8805","html_url":"https://github.com/apache/spark/pull/8805","diff_url":"https://github.com/apache/spark/pull/8805.diff","patch_url":"https://github.com/apache/spark/pull/8805.patch"},"body":"Currently `UnsafeExternalSorter` acquires memory like this:\r\n(1) Request memory for a task\r\n(2) If (1) fails, spill and release all of our memory\r\n(3) Try request memory for a task again\r\n(4) If (3) fails, throw exception\r\n\r\nBecause we release all of our memory, however, we introduce the chance for other tasks to jump in and steal our memory allocation, which causes issues like SPARK-10474. Instead, we should just release a fraction of the memory and keep the ones that we *know* we're going to use immediately afterwards.\r\n\r\nWIP because tests are coming."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/141534690","html_url":"https://github.com/apache/spark/pull/8805#issuecomment-141534690","issue_url":"https://api.github.com/repos/apache/spark/issues/8805","id":141534690,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T18:52:56Z","updated_at":"2015-09-18T18:52:56Z","body":"Actually @yhuai and I just found the real root cause of the original issue SPARK-10474. I'm closing this PR."}}