{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/1977","labels_url":"https://api.github.com/repos/apache/spark/issues/1977/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/1977/comments","events_url":"https://api.github.com/repos/apache/spark/issues/1977/events","html_url":"https://github.com/apache/spark/pull/1977","id":40398443,"number":1977,"title":"[SPARK-3074] [PySpark] support groupByKey() with single huge key","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":110,"created_at":"2014-08-16T00:32:33Z","updated_at":"2015-04-06T22:03:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/1977","html_url":"https://github.com/apache/spark/pull/1977","diff_url":"https://github.com/apache/spark/pull/1977.diff","patch_url":"https://github.com/apache/spark/pull/1977.patch"},"body":"This patch change groupByKey() to use external sort based approach, so it can support single huge key.\r\n\r\nFor example, it can group by a dataset including one hot key with 40 millions values (strings), using 500M memory for Python worker, finished in about 2 minutes. (it will need 6G memory in hash based approach).\r\n\r\nDuring groupByKey(), it will do in-memory groupBy first. If the dataset can not fit in memory, then data will be partitioned by hash. If one partition still can not fit in memory, it will switch to sort based groupBy(). "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/90258222","html_url":"https://github.com/apache/spark/pull/1977#issuecomment-90258222","issue_url":"https://api.github.com/repos/apache/spark/issues/1977","id":90258222,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-04-06T22:03:23Z","updated_at":"2015-04-06T22:03:23Z","body":"  [Test build #29757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29757/consoleFull) for   PR 1977 at commit [`d9589ab`](https://github.com/apache/spark/commit/d9589ab0dbcd6aa443906c3b1ec432d6c68f0587)."}}