{"action":"created","issue":{"url":"https://api.github.com/repos/mrocklin/distributed3/issues/1","labels_url":"https://api.github.com/repos/mrocklin/distributed3/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/mrocklin/distributed3/issues/1/comments","events_url":"https://api.github.com/repos/mrocklin/distributed3/issues/1/events","html_url":"https://github.com/mrocklin/distributed3/pull/1","id":109169887,"number":1,"title":"Dask","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-30T20:12:57Z","updated_at":"2015-10-05T15:08:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mrocklin/distributed3/pulls/1","html_url":"https://github.com/mrocklin/distributed3/pull/1","diff_url":"https://github.com/mrocklin/distributed3/pull/1.diff","patch_url":"https://github.com/mrocklin/distributed3/pull/1.patch"},"body":"This is mostly a play project.  The goal here is to see how simple we can make writing a reasonable dask.distributed scheduler.  This implementation relies *very heavily* on logic within the tornado event loop but manages a nearly complete implementation in a bit more than 200 lines.\r\n\r\nDon't expect much.  In particular questions of the form \"But will this handle X?\" can probably be answered with \"no.\"\r\n\r\nThis manages to be less code than the single machine scheduler because it depends on the setup in the hosting `distributed3` project (also a play project) and because it spins up a future for every task and relies on the tornado event loop to figure out when these tasks should run.  I'm not sure exactly how well the tornado event loop is built and if it can handle millions of little `Event` objects; this implementation may not scale well. \r\n\r\nThis can get around 800 tasks per second on my low-powered laptop.\r\n\r\nExample\r\n--------\r\n\r\n### Setup local cluster\r\n\r\n    $ bin/dcenter                           # Center\r\n    Started center at 127.0.0.1:8787\r\n\r\n    $ bin/dworker 127.0.0.1:8787            # Worker 1\r\n    Start worker at 127.0.0.1:8788\r\n    Register with center at 127.0.0.1:8787\r\n\r\n    $ bin/dworker 127.0.0.1:8787            # Worker 2\r\n    Start worker at 127.0.0.1:8789\r\n    Register with center at 127.0.0.1:8787\r\n\r\n### Use distributed3 get function\r\n\r\n```python\r\nIn [1]: from distributed3.dask import get\r\n\r\nIn [2]: dsk = {'x': 1, 'y': (lambda x: x + 1, 'x')}\r\n\r\nIn [3]: get('127.0.0.1', 8787, dsk, 'y')\r\nOut[3]: RemoteData<center=127.0.0.1:8787, key=y>\r\n\r\nIn [4]: _.get()\r\nOut[4]: 2\r\n```\r\n\r\n### Use with collections\r\n\r\n```python\r\nIn [1]: from distributed3.dask import get\r\n\r\nIn [2]: from functools import partial\r\n\r\nIn [3]: get2 = partial(get, '127.0.0.1', 8787, gather=True)\r\n\r\nIn [4]: import dask.array as da\r\n\r\nIn [5]: x = da.ones((1000, 1000), chunks=(100, 100))\r\n\r\nIn [6]: x.sum(axis=1).compute(get=get2)\r\nOut[6]: \r\narray([ 1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,\r\n        1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,\r\n        1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,\r\n        1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,\r\n        1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,  1000.,\r\n        ...\r\n```\r\n\r\nThis is just up here for general commenting.  Questions/comments quite welcome."},"comment":{"url":"https://api.github.com/repos/mrocklin/distributed3/issues/comments/145563241","html_url":"https://github.com/mrocklin/distributed3/pull/1#issuecomment-145563241","issue_url":"https://api.github.com/repos/mrocklin/distributed3/issues/1","id":145563241,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-10-05T15:08:00Z","updated_at":"2015-10-05T15:08:00Z","body":"This is good enough to be used.  I'm thinking about replacing it with something a bit less dependent on tornado event loop.  Merging now."}}