{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5463","labels_url":"https://api.github.com/repos/apache/spark/issues/5463/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5463/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5463/events","html_url":"https://github.com/apache/spark/pull/5463","id":67715872,"number":5463,"title":"[Spark-6839] BlockManger.dataDeserialize must ensure input stream is closed even on exceptions","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":60,"created_at":"2015-04-11T01:19:18Z","updated_at":"2015-07-28T16:16:17Z","closed_at":"2015-07-28T16:16:17Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5463","html_url":"https://github.com/apache/spark/pull/5463","diff_url":"https://github.com/apache/spark/pull/5463.diff","patch_url":"https://github.com/apache/spark/pull/5463.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-6839\r\n\r\nThis is fixing a potential resource leak when reading from the cache and when reading shuffle data from ExternalSorter.\r\n\r\nThis needed to touch a surprisingly large amount of code to make sure that `BlockManager.dataDeserialize` always gets passed something which can ensure the input stream gets closed.  I trimmed out what I could, but almost all paths through `BlockManager` might end up calling `dataDeserialize` (even when blocks are being `put`).\r\n\r\nThere isn't always a `TaskContext` at all of the relevant call sites, so I made a new abstraction `ResourceCleaner`.  `TaskContext` extends `ResourceCleaner`, so we use `TaskContext` where we have one; otherwise there is a `SimpleResourceCleaner` that just keeps a list of functions to run in a `finally` block.\r\n\r\nI also considered forcing `DeserializationStream.asIterator` to need  a `ResourceCleaner`.  That way we'd force the *right* cleaning function to be used everytime somebody called `stream.asIterator`.  However, I figure it is *possible* that you might read some of the stream and not necessarily want close it.  I'm curious for the opinion of others on this.\r\n\r\n`BroadcastSuite` is somewhat flaky when run on my laptop ... I'm pretty sure that is not related to these changes, but I guess we'll see what jenkins says."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/125667585","html_url":"https://github.com/apache/spark/pull/5463#issuecomment-125667585","issue_url":"https://api.github.com/repos/apache/spark/issues/5463","id":125667585,"user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"created_at":"2015-07-28T16:16:17Z","updated_at":"2015-07-28T16:16:17Z","body":"sorry forgot to get back to this.  Sure, I'll close.  I think its not ideal that our leak-detector is just waiting for users to complain, but I also have no idea how important this is."}}