{"action":"created","issue":{"url":"https://api.github.com/repos/diatomic/test/issues/1","labels_url":"https://api.github.com/repos/diatomic/test/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/diatomic/test/issues/1/comments","events_url":"https://api.github.com/repos/diatomic/test/issues/1/events","html_url":"https://github.com/diatomic/test/issues/1","id":79666086,"number":1,"title":"Move queues out of core","user":{"login":"mrzv","id":720816,"avatar_url":"https://avatars.githubusercontent.com/u/720816?v=3","gravatar_id":"","url":"https://api.github.com/users/mrzv","html_url":"https://github.com/mrzv","followers_url":"https://api.github.com/users/mrzv/followers","following_url":"https://api.github.com/users/mrzv/following{/other_user}","gists_url":"https://api.github.com/users/mrzv/gists{/gist_id}","starred_url":"https://api.github.com/users/mrzv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrzv/subscriptions","organizations_url":"https://api.github.com/users/mrzv/orgs","repos_url":"https://api.github.com/users/mrzv/repos","events_url":"https://api.github.com/users/mrzv/events{/privacy}","received_events_url":"https://api.github.com/users/mrzv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mrzv","id":720816,"avatar_url":"https://avatars.githubusercontent.com/u/720816?v=3","gravatar_id":"","url":"https://api.github.com/users/mrzv","html_url":"https://github.com/mrzv","followers_url":"https://api.github.com/users/mrzv/followers","following_url":"https://api.github.com/users/mrzv/following{/other_user}","gists_url":"https://api.github.com/users/mrzv/gists{/gist_id}","starred_url":"https://api.github.com/users/mrzv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrzv/subscriptions","organizations_url":"https://api.github.com/users/mrzv/orgs","repos_url":"https://api.github.com/users/mrzv/repos","events_url":"https://api.github.com/users/mrzv/events{/privacy}","received_events_url":"https://api.github.com/users/mrzv/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-05-23T00:21:36Z","updated_at":"2015-05-23T00:28:10Z","closed_at":null,"body":"**Issue by [Dmitriy Morozov](https://bitbucket.com/mrzv)**\n_Friday Nov 14, 2014 at 17:51 GMT_\n\n----\n\nAdd the ability to move queues out of core (when their blocks are also out of core). For example, this is essential to support IO-efficient distribution of points (through a swap-reduce). Roughly, the protocol should be:\r\n\r\n\r\n```\r\n#!text\r\n\r\nforeach block:\r\n  process(block)\r\n  exchange-light   # optional\r\n\r\nexchange-heavy\r\n\r\nexchange-light:\r\n  post queues\r\n  kick outstanding messages\r\n  if #queues too large:\r\n    swap some queues out\r\n\r\nexchange-heavy:\r\n  post any unposted queues   # happens if exchange-light wasn't invoked\r\n  kick outstanding messages\r\n  while there are external outgoing queues:\r\n    if #posted queues < #allowed queues:\r\n      load and post more queues\r\n    kick outstanding messages\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/diatomic/test/issues/comments/104806617","html_url":"https://github.com/diatomic/test/issues/1#issuecomment-104806617","issue_url":"https://api.github.com/repos/diatomic/test/issues/1","id":104806617,"user":{"login":"mrzv","id":720816,"avatar_url":"https://avatars.githubusercontent.com/u/720816?v=3","gravatar_id":"","url":"https://api.github.com/users/mrzv","html_url":"https://github.com/mrzv","followers_url":"https://api.github.com/users/mrzv/followers","following_url":"https://api.github.com/users/mrzv/following{/other_user}","gists_url":"https://api.github.com/users/mrzv/gists{/gist_id}","starred_url":"https://api.github.com/users/mrzv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrzv/subscriptions","organizations_url":"https://api.github.com/users/mrzv/orgs","repos_url":"https://api.github.com/users/mrzv/repos","events_url":"https://api.github.com/users/mrzv/events{/privacy}","received_events_url":"https://api.github.com/users/mrzv/received_events","type":"User","site_admin":false},"created_at":"2015-05-23T00:28:10Z","updated_at":"2015-05-23T00:28:10Z","body":"**Comment by [Dmitriy Morozov](https://bitbucket.com/mrzv)**\n_Wednesday Nov 26, 2014 at 21:35 GMT_\n\n----\n\nThis does not address what to do with incoming queues. These need more thought. Most likely, we have to re-engineer `Communicator` and `Master`. Specifically, `Master` might have to own incoming and outgoing queues and transparently to `Communicator` decide when they need to be moved in or out of core."}}