{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8827","labels_url":"https://api.github.com/repos/apache/spark/issues/8827/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8827/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8827/events","html_url":"https://github.com/apache/spark/pull/8827","id":107270653,"number":8827,"title":"[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-18T20:20:15Z","updated_at":"2015-09-18T20:29:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8827","html_url":"https://github.com/apache/spark/pull/8827","diff_url":"https://github.com/apache/spark/pull/8827.diff","patch_url":"https://github.com/apache/spark/pull/8827.patch"},"body":"When `TungstenAggregation` hits memory pressure, it switches from hash-based to sort-based aggregation in-place. However, in the process we try to allocate the pointer array for writing to the new `UnsafeExternalSorter` *before* actually freeing the memory from the hash map. This lead to the following exception:\r\n```\r\n java.io.IOException: Could not acquire 65536 bytes of memory\r\n        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.initializeForWriting(UnsafeExternalSorter.java:169)\r\n        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:220)\r\n        at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:126)\r\n        at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:257)\r\n        at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.switchToSortBasedAggregation(TungstenAggregationIterator.scala:435)\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/141560514","html_url":"https://github.com/apache/spark/pull/8827#issuecomment-141560514","issue_url":"https://api.github.com/repos/apache/spark/issues/8827","id":141560514,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T20:29:54Z","updated_at":"2015-09-18T20:29:54Z","body":"  [Test build #42684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42684/consoleFull) for   PR 8827 at commit [`f7dd17f`](https://github.com/apache/spark/commit/f7dd17fca10d33e5afceb9df934d5a2f2290abae)."}}