{"action":"created","issue":{"url":"https://api.github.com/repos/ContinuumIO/dask/issues/489","labels_url":"https://api.github.com/repos/ContinuumIO/dask/issues/489/labels{/name}","comments_url":"https://api.github.com/repos/ContinuumIO/dask/issues/489/comments","events_url":"https://api.github.com/repos/ContinuumIO/dask/issues/489/events","html_url":"https://github.com/ContinuumIO/dask/pull/489","id":97970825,"number":489,"title":"[WIP] Reuse Distributed State","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-29T16:06:28Z","updated_at":"2015-07-29T19:45:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ContinuumIO/dask/pulls/489","html_url":"https://github.com/ContinuumIO/dask/pull/489","diff_url":"https://github.com/ContinuumIO/dask/pull/489.diff","patch_url":"https://github.com/ContinuumIO/dask/pull/489.patch"},"body":"This is a WIP PR to establish and reuse state in the distributed scheduler. \r\n\r\nWhen computing on the distributed scheduler you can provide a keyword argument to keep the results in memory on the workers.  When future computations occur they reuse this state.\r\n\r\n- [x] Execution state checks provided cache\r\n- [x] `distributed.Scheduler.schedule` culls unnecessary parts of the graph after looking at data in current memory\r\n- [ ] Provide user hook to ask for the result to stay in memory rather than be cleared out.\r\n\r\nFixes #488 cc @dataisle"},"comment":{"url":"https://api.github.com/repos/ContinuumIO/dask/issues/comments/126075221","html_url":"https://github.com/ContinuumIO/dask/pull/489#issuecomment-126075221","issue_url":"https://api.github.com/repos/ContinuumIO/dask/issues/489","id":126075221,"user":{"login":"dataisle","id":3936122,"avatar_url":"https://avatars.githubusercontent.com/u/3936122?v=3","gravatar_id":"","url":"https://api.github.com/users/dataisle","html_url":"https://github.com/dataisle","followers_url":"https://api.github.com/users/dataisle/followers","following_url":"https://api.github.com/users/dataisle/following{/other_user}","gists_url":"https://api.github.com/users/dataisle/gists{/gist_id}","starred_url":"https://api.github.com/users/dataisle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dataisle/subscriptions","organizations_url":"https://api.github.com/users/dataisle/orgs","repos_url":"https://api.github.com/users/dataisle/repos","events_url":"https://api.github.com/users/dataisle/events{/privacy}","received_events_url":"https://api.github.com/users/dataisle/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T19:45:24Z","updated_at":"2015-07-29T19:45:24Z","body":"Whats the reason for removing the cache after the key has been read? e.g.:\r\n\r\n```\r\ns.send_data('x',10,address=w.address)\r\nw.data['x']\r\nOut[18]: 10\r\ns.schedule({'x': (inc, 1), 'y': (lambda x: x, 'x')}, 'y') == 10\r\nOut[19]: True\r\nw.data['x']\r\n---------------------------------------------------------------------------\r\nKeyError                                  Traceback (most recent call last)\r\n<ipython-input-17-2c01e83b2f40> in <module>()\r\n----> 1 w.data['x']\r\n\r\nKeyError: 'x'\r\n\r\n```"}}