{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4923","labels_url":"https://api.github.com/repos/apache/spark/issues/4923/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4923/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4923/events","html_url":"https://github.com/apache/spark/pull/4923","id":60048928,"number":4923,"title":"[SPARK-6194] [PySpark] fix memory leak in collect()","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-03-06T02:15:31Z","updated_at":"2015-03-09T01:00:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4923","html_url":"https://github.com/apache/spark/pull/4923","diff_url":"https://github.com/apache/spark/pull/4923.diff","patch_url":"https://github.com/apache/spark/pull/4923.patch"},"body":"Because circular reference between JavaObject and JavaMember, an Java object can not be released until Python GC kick in, then it will cause memory leak in collect(), which may consume lots of memory in JVM.\r\n\r\nThis PR change the collect() to get the iterator from JVM directly (avoid a method call `iterator()`), it also fix a bug in py4j, which will create a `Java` member for all the JavaObject parameters.\r\n\r\ncc @JoshRosen "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/77786229","html_url":"https://github.com/apache/spark/pull/4923#issuecomment-77786229","issue_url":"https://api.github.com/repos/apache/spark/issues/4923","id":77786229,"user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"created_at":"2015-03-09T01:00:19Z","updated_at":"2015-03-09T01:00:19Z","body":"This looks really good to me overall.  It would be great if you could update the PR description to reflect the most recent changes (collecting via a socket instead of a temporary file).\r\n\r\nA couple of quick questions:\r\n\r\n- Does this have a performance impact (positive or negative)?\r\n- Do we need to write any additional tests?  Are there any corner cases that we should explicitly address in their own tests, such as not consuming the iterator or not connecting to the socket after creating it?"}}