{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5403","labels_url":"https://api.github.com/repos/apache/spark/issues/5403/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5403/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5403/events","html_url":"https://github.com/apache/spark/pull/5403","id":66992855,"number":5403,"title":"[SPARK-3376] Add in-memory shuffle option.","user":{"login":"kayousterhout","id":1108612,"avatar_url":"https://avatars.githubusercontent.com/u/1108612?v=3","gravatar_id":"","url":"https://api.github.com/users/kayousterhout","html_url":"https://github.com/kayousterhout","followers_url":"https://api.github.com/users/kayousterhout/followers","following_url":"https://api.github.com/users/kayousterhout/following{/other_user}","gists_url":"https://api.github.com/users/kayousterhout/gists{/gist_id}","starred_url":"https://api.github.com/users/kayousterhout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayousterhout/subscriptions","organizations_url":"https://api.github.com/users/kayousterhout/orgs","repos_url":"https://api.github.com/users/kayousterhout/repos","events_url":"https://api.github.com/users/kayousterhout/events{/privacy}","received_events_url":"https://api.github.com/users/kayousterhout/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-04-07T21:17:36Z","updated_at":"2015-04-20T23:02:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5403","html_url":"https://github.com/apache/spark/pull/5403","diff_url":"https://github.com/apache/spark/pull/5403.diff","patch_url":"https://github.com/apache/spark/pull/5403.patch"},"body":"This commit adds a new ShuffleManager that stores all shuffle data\r\nin-memory using the block manager.\r\n\r\nWith this change, in-memory shuffle data is cleaned up in the\r\nsame way as disk shuffle data: using the metadata cleaner. It\r\nwould probably be better to clean up in-memory shuffle data more\r\naggressively, in order to avoid running out of memory.\r\n\r\nOne idea proposed by @pwendell was to publish this as a Spark\r\npackage rather than adding it directly to Spark. The concern is that\r\nthis option may be confusing to naive users, because it can result in\r\nSpark's memory filling up in unexpected ways.  If it is decided that\r\nthis should actually be added to Spark, I can update the documentation\r\nto show the in-memory shuffle as one of the possible shuffle managers.\r\n\r\ncc @shivaram "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/94588226","html_url":"https://github.com/apache/spark/pull/5403#issuecomment-94588226","issue_url":"https://api.github.com/repos/apache/spark/issues/5403","id":94588226,"user":{"login":"pwendell","id":320616,"avatar_url":"https://avatars.githubusercontent.com/u/320616?v=3","gravatar_id":"","url":"https://api.github.com/users/pwendell","html_url":"https://github.com/pwendell","followers_url":"https://api.github.com/users/pwendell/followers","following_url":"https://api.github.com/users/pwendell/following{/other_user}","gists_url":"https://api.github.com/users/pwendell/gists{/gist_id}","starred_url":"https://api.github.com/users/pwendell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwendell/subscriptions","organizations_url":"https://api.github.com/users/pwendell/orgs","repos_url":"https://api.github.com/users/pwendell/repos","events_url":"https://api.github.com/users/pwendell/events{/privacy}","received_events_url":"https://api.github.com/users/pwendell/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T23:02:57Z","updated_at":"2015-04-20T23:02:57Z","body":"Since it's a pretty simple implementation, I'd be fine if it were merged in. But I think we should say clearly that it can be useful for benchmarking, etc, but isn't meant to be used in production setting since it's not robust to OOM. /cc @rxin for his thoughts also."}}