{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/1977","labels_url":"https://api.github.com/repos/apache/spark/issues/1977/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/1977/comments","events_url":"https://api.github.com/repos/apache/spark/issues/1977/events","html_url":"https://github.com/apache/spark/pull/1977","id":40398443,"number":1977,"title":"[SPARK-3074] [PySpark] support groupByKey() with single huge key","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":119,"created_at":"2014-08-16T00:32:33Z","updated_at":"2015-04-07T18:57:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/1977","html_url":"https://github.com/apache/spark/pull/1977","diff_url":"https://github.com/apache/spark/pull/1977.diff","patch_url":"https://github.com/apache/spark/pull/1977.patch"},"body":"This patch change groupByKey() to use external sort based approach, so it can support single huge key.\r\n\r\nFor example, it can group by a dataset including one hot key with 40 millions values (strings), using 500M memory for Python worker, finished in about 2 minutes. (it will need 6G memory in hash based approach).\r\n\r\nDuring groupByKey(), it will do in-memory groupBy first. If the dataset can not fit in memory, then data will be partitioned by hash. If one partition still can not fit in memory, it will switch to sort based groupBy(). "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/90697704","html_url":"https://github.com/apache/spark/pull/1977#issuecomment-90697704","issue_url":"https://api.github.com/repos/apache/spark/issues/1977","id":90697704,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T18:57:22Z","updated_at":"2015-04-07T18:57:22Z","body":"  [Test build #29802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29802/consoleFull) for   PR 1977 at commit [`2a1857a`](https://github.com/apache/spark/commit/2a1857a4c1d72bee5a93ffbe1da013ffc7ff673b).\n * This patch **fails PySpark unit tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `class ResultIterable(object):`\n  * `class FlattedValuesSerializer(BatchedSerializer):`\n  * `class ExternalList(object):`\n  * `class GroupByKey(object):`\n  * `class ChainedIterable(object):`\n  * `class ExternalGroupBy(ExternalMerger):`\n\n * This patch does not change any dependencies."}}