{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5685","labels_url":"https://api.github.com/repos/apache/spark/issues/5685/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5685/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5685/events","html_url":"https://github.com/apache/spark/pull/5685","id":70649496,"number":5685,"title":"[SPARK-7120][SPARK-7121] Closure cleaner nesting + documentation + tests","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2015-04-24T10:19:21Z","updated_at":"2015-04-29T02:37:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5685","html_url":"https://github.com/apache/spark/pull/5685","diff_url":"https://github.com/apache/spark/pull/5685.diff","patch_url":"https://github.com/apache/spark/pull/5685.patch"},"body":"Note: ~600 lines of this is test code, and ~100 lines documentation.\r\n\r\n**[SPARK-7121]** ClosureCleaner does not handle nested closures properly. For instance, in SparkContext, I tried to do the following:\r\n```\r\ndef scope[T](body: => T): T = body // no-op\r\ndef myCoolMethod(path: String): RDD[String] = scope {\r\n  parallelize(1 to 10).map { _ => path }\r\n}\r\n```\r\nand I got an exception complaining that SparkContext is not serializable. The issue here is that the inner closure is getting its path from the outer closure (the scope), but the outer closure references the SparkContext object itself to get the `parallelize` method.\r\n\r\nNote, however, that the inner closure doesn't actually need the SparkContext; it just needs a field from the outer closure. If we modify ClosureCleaner to clean the outer closure recursively using only the fields accessed by the inner closure, then we can serialize the inner closure.\r\n\r\n**[SPARK-7120]** Also, the other thing is that this file is one of the least understood, partly because it is very low level and is written a long time ago. This patch attempts to change that by adding the missing documentation.\r\n\r\nThis is blocking my effort on a separate task #5729."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/97287023","html_url":"https://github.com/apache/spark/pull/5685#issuecomment-97287023","issue_url":"https://api.github.com/repos/apache/spark/issues/5685","id":97287023,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T02:37:48Z","updated_at":"2015-04-29T02:37:48Z","body":"  [Test build #31217 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31217/consoleFull) for   PR 5685 at commit [`e909a42`](https://github.com/apache/spark/commit/e909a4222a132b1fc4fff87a111ece7d32b82765)."}}