{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4827","labels_url":"https://api.github.com/repos/apache/spark/issues/4827/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4827/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4827/events","html_url":"https://github.com/apache/spark/pull/4827","id":59329513,"number":4827,"title":"[SPARK-6076][Block Manager] Fix a potential OOM issue when StorageLevel is MEMORY_AND_DISK_SER","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-02-28T06:51:20Z","updated_at":"2015-03-21T03:44:31Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4827","html_url":"https://github.com/apache/spark/pull/4827","diff_url":"https://github.com/apache/spark/pull/4827.diff","patch_url":"https://github.com/apache/spark/pull/4827.patch"},"body":"In https://github.com/apache/spark/blob/dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L538 , when StorageLevel is `MEMORY_AND_DISK_SER`, it will copy the content from file into memory, then put it into MemoryStore.\r\n```scala\r\n              val copyForMemory = ByteBuffer.allocate(bytes.limit)\r\n              copyForMemory.put(bytes)\r\n              memoryStore.putBytes(blockId, copyForMemory, level)\r\n              bytes.rewind()\r\n```\r\nHowever, if the file is bigger than the free memory, OOM will happen. A better approach is testing if there is enough memory. If not, copyForMemory should not be created, since this is an optional operation."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/84238397","html_url":"https://github.com/apache/spark/pull/4827#issuecomment-84238397","issue_url":"https://api.github.com/repos/apache/spark/issues/4827","id":84238397,"user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"created_at":"2015-03-21T03:44:31Z","updated_at":"2015-03-21T03:44:31Z","body":"@andrewor14 is it acceptable that changing call-by-name to `() => Unit`?"}}