{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/44","labels_url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/44/comments","events_url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/44/events","html_url":"https://github.com/JuliaParallel/DistributedArrays.jl/pull/44","id":112635848,"number":44,"title":"WIP/RFC: Remove use of RemoteRefs","user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-10-21T17:05:24Z","updated_at":"2015-10-26T11:33:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/pulls/44","html_url":"https://github.com/JuliaParallel/DistributedArrays.jl/pull/44","diff_url":"https://github.com/JuliaParallel/DistributedArrays.jl/pull/44.diff","patch_url":"https://github.com/JuliaParallel/DistributedArrays.jl/pull/44.patch"},"body":"The use of RemoteRefs and the existing distributed gc implementation results in poor scaling when talking about 100s or 1000s of workers. For example, serializing a DArray distributed across 1000 workers will result in almost a million DGC messages.\r\n\r\nThis PR eschews the use of RemoteRefs in favour of a module level dictionary to associate \"localparts\" of the array. The changes are:\r\n\r\n- every DArray has a unique `identity` comprising of the tuple (pid_where_created, one_up_number).\r\n- module level dictionary called `registry` which has two mappings for every darray\r\n  - (identity, :LOCALPART) -> localpart of the array\r\n  - (identity, :DARRAY) -> the darrays metadata - pids, cuts, indexes, etc.\r\n- Only the `identity` tuple is sent across the network while serializing/deserializing.\r\n- Distributed memory can be explicitly released by calling `close(::DArray)` on the constructing process. This aids in the distributed memory being released immediately instead of waiting for the darray object to be collected on the constructing process (which today is unpredictable due to a lack of memory pressure)\r\n- Not calling `close` results in the DArray being released when the object is collected as before. `close` is optional.\r\n\r\n#### Limitations from existing implementation\r\n\r\n- The DArray is released when the first reference to the darray is collected. The programmer thus has to ensure that the first handle to the DArray on the constructing process remains in scope for the duration of the computation. \r\n- Only the creating process and participating workers have full access to the DArray metadata. This is because the metadata is set upon construction and serialization of a DArray object only sends the identity tuple.\r\n\r\n#### TODO\r\n- [ ] - Benchmark current implementation vs this PR on 1000 workers\r\n- [ ] - documentation updates\r\n   \r\n"},"comment":{"url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/comments/151106457","html_url":"https://github.com/JuliaParallel/DistributedArrays.jl/pull/44#issuecomment-151106457","issue_url":"https://api.github.com/repos/JuliaParallel/DistributedArrays.jl/issues/44","id":151106457,"user":{"login":"ViralBShah","id":744411,"avatar_url":"https://avatars.githubusercontent.com/u/744411?v=3","gravatar_id":"","url":"https://api.github.com/users/ViralBShah","html_url":"https://github.com/ViralBShah","followers_url":"https://api.github.com/users/ViralBShah/followers","following_url":"https://api.github.com/users/ViralBShah/following{/other_user}","gists_url":"https://api.github.com/users/ViralBShah/gists{/gist_id}","starred_url":"https://api.github.com/users/ViralBShah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ViralBShah/subscriptions","organizations_url":"https://api.github.com/users/ViralBShah/orgs","repos_url":"https://api.github.com/users/ViralBShah/repos","events_url":"https://api.github.com/users/ViralBShah/events{/privacy}","received_events_url":"https://api.github.com/users/ViralBShah/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T11:33:25Z","updated_at":"2015-10-26T11:33:25Z","body":"+1 to generalising the registry. Distributed graphs and hash tables should serve as good test cases."}}