{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/855","labels_url":"https://api.github.com/repos/blaze/dask/issues/855/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/855/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/855/events","html_url":"https://github.com/blaze/dask/issues/855","id":119587410,"number":855,"title":"parallel optimization feedback","user":{"login":"ajfriend","id":984039,"avatar_url":"https://avatars.githubusercontent.com/u/984039?v=3","gravatar_id":"","url":"https://api.github.com/users/ajfriend","html_url":"https://github.com/ajfriend","followers_url":"https://api.github.com/users/ajfriend/followers","following_url":"https://api.github.com/users/ajfriend/following{/other_user}","gists_url":"https://api.github.com/users/ajfriend/gists{/gist_id}","starred_url":"https://api.github.com/users/ajfriend/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajfriend/subscriptions","organizations_url":"https://api.github.com/users/ajfriend/orgs","repos_url":"https://api.github.com/users/ajfriend/repos","events_url":"https://api.github.com/users/ajfriend/events{/privacy}","received_events_url":"https://api.github.com/users/ajfriend/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-30T21:42:15Z","updated_at":"2015-11-30T23:50:34Z","closed_at":null,"body":"@mrocklin was helping me implement a parallel optimization algorithm in dask. Here's the resulting notebook of a first naive implementation, along with some notes and bugs I was having. @mrocklin mentioned it might be useful to others.\r\n\r\nRight now, I'm wondering why I'm not seeing a performance gain with the threaded scheduler (I may just be missing something obvious). \r\n\r\nI was also getting many runtime errors as I was switching between schedulers and methods of setting the schedulers. I've left two of those tracebacks in the notebook.\r\n\r\nhttps://gist.github.com/ajfriend/d6324d1e81ab04dbb0fd"},"comment":{"url":"https://api.github.com/repos/blaze/dask/issues/comments/160801403","html_url":"https://github.com/blaze/dask/issues/855#issuecomment-160801403","issue_url":"https://api.github.com/repos/blaze/dask/issues/855","id":160801403,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-11-30T23:50:34Z","updated_at":"2015-11-30T23:50:34Z","body":"Hrm, the `num_workers` thing is interesting.  Maybe we should retain ThreadPool objects as some state here, keeping around a dictionary mapping `num_workers` to the appropriate `ThreadPool`\r\n\r\n```python\r\nthread_pools = dict()\r\n\r\ndef get_threadpool(num_workers):\r\n    if num_workers not in thread_pools:\r\n        thread_pools[num_workers] = ThreadPool(num_workers)\r\n    return thread_pools[num_workers]\r\n```\r\n\r\nOr, if we wanted to be hip (and add a dependency that we shouldn't)\r\n\r\n```python\r\nthreadpool = toolz.memoize(ThreadPool, cache=thread_pools)\r\n```"}}