{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8011","labels_url":"https://api.github.com/repos/apache/spark/issues/8011/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8011/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8011/events","html_url":"https://github.com/apache/spark/pull/8011","id":99538855,"number":8011,"title":"[SPARK-9709] [SQL] Avoid starving unsafe operators that use sort","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-08-06T22:33:55Z","updated_at":"2015-08-06T23:42:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8011","html_url":"https://github.com/apache/spark/pull/8011","diff_url":"https://github.com/apache/spark/pull/8011.diff","patch_url":"https://github.com/apache/spark/pull/8011.patch"},"body":"The issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an `IOException(\"Unable to acquire X bytes of memory\")` thrown by `UnsafeExternalSorter`.\r\n\r\nThe solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/128540296","html_url":"https://github.com/apache/spark/pull/8011#issuecomment-128540296","issue_url":"https://api.github.com/repos/apache/spark/issues/8011","id":128540296,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-06T23:42:50Z","updated_at":"2015-08-06T23:42:50Z","body":" Merged build triggered."}}