{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/13923","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/13923/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/13923/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/13923/events","html_url":"https://github.com/JuliaLang/julia/pull/13923","id":115867145,"number":13923,"title":"Updating RemoteRefs to Futures and RemoteChannels","user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/parallel","name":"parallel","color":"f7e102"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-11-09T12:25:00Z","updated_at":"2015-11-12T04:31:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/13923","html_url":"https://github.com/JuliaLang/julia/pull/13923","diff_url":"https://github.com/JuliaLang/julia/pull/13923.diff","patch_url":"https://github.com/JuliaLang/julia/pull/13923.patch"},"body":"This PR is ready for a review:\r\n\r\nThe intention of this PR is to optimize the messaging overhead w.r.t. the current `RemoteRef` model. \r\n\r\nA `Future` is a write-once remote reference and all remote calls now return `Future`s. The existing `RemoteRef` has been renamed to `RemoteChannel`, since in effect, a `RemoteChannel` points to an implementation of a `AbstractChannel` on a remote node.\r\n\r\nBoth `Future` and `RemoteChannel` are implementations of `AbstractRemoteRef`. \r\n\r\nThe below section is the same as the updated manual in this PR: \r\n\r\nRemote References and Distributed Garbage Collection\r\n----------------------------------------------------\r\n\r\nObjects referred to by remote references can be freed only when *all* held references in the cluster\r\nare deleted.\r\n\r\nThe node where the value is stored keeps tracks of which of the workers have a reference to it.\r\nEvery time a ``RemoteChannel`` or a (unfetched) ``Future`` is serialized to a worker, the node pointed\r\nto by the reference is notified. And every time a ``RemoteChannel`` or a (unfetched) ``Future``\r\nis garbage collected locally, the node owning the value is again notified.\r\n\r\nThe notifications are done via sending of \"tracking\" messages - an \"add reference\" message when\r\na reference is serialized to a different process and a \"delete reference\" message when a reference\r\nis locally garbage collected.\r\n\r\nSince ``Future``\\ s are write-once and cached locally, the act of ``fetch``ing a ``Future`` also updates\r\nreference tracking information on the node owning the value.\r\n\r\nThe node which owns the value, frees it once all references to it are cleared.\r\n\r\nWith ``Future``\\ s, serializing an already fetched ``Future`` to a different node also sends the value\r\nsince the original remote store may have collected the value by this time.\r\n\r\nIt is important to note that *when* an object is locally garbage collected is dependent\r\nupon the size of the object and the current memory pressure in the system.\r\n\r\nIn case of remote references, the size of the local reference object is quite small, while the remote value\r\nstored on the remote node may be quite large. Since the local object may not be collected immediately, it is\r\na good practice to explicitly call ``finalize`` on local instances of a ``RemoteChannel``, or on unfetched\r\n``Future``\\ s. Since calling ``fetch`` on a ``Future`` also removes its reference from the remote store, this\r\nis not required on fetched ``Future``\\ s. Explicitly calling ``finalize`` immediately sends a message to\r\nthe remote node to go ahead and remove its reference to the value.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/155994823","html_url":"https://github.com/JuliaLang/julia/pull/13923#issuecomment-155994823","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/13923","id":155994823,"user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T04:31:27Z","updated_at":"2015-11-12T04:31:27Z","body":"For \r\n\r\n```\r\naddprocs(1)\r\n\r\n@everywhere echo(x) = x\r\n\r\nfunction foo(n)\r\n    for i in 1:10^n\r\n        f=remotecall(echo, 2, 1)\r\n        fetch(f)\r\n    end\r\nend\r\n\r\nfoo(1)\r\n@time foo(5)\r\n```\r\n\r\nOn current master:\r\n\r\n```\r\njulia> @time foo(5)\r\n 14.342809 seconds (19.87 M allocations: 1.115 GB, 1.85% gc time)\r\n\r\njulia> @time foo(5)\r\n 14.446782 seconds (19.88 M allocations: 1.115 GB, 2.09% gc time)\r\n```\r\n\r\nWith this PR:\r\n\r\n```\r\njulia> @time foo(5)\r\n 10.433722 seconds (18.25 M allocations: 1.006 GB, 1.66% gc time)\r\n\r\njulia> @time foo(5)\r\n  9.475284 seconds (18.25 M allocations: 1.006 GB, 1.90% gc time)\r\n```\r\n\r\nThis is probably due to the absence of \"del reference \" messages in this test case.  \r\n\r\n"}}