{"action":"created","comment":{"url":"https://api.github.com/repos/ContinuumIO/dask/pulls/comments/33522073","id":33522073,"diff_hunk":"@@ -0,0 +1,86 @@\n+Diagnostics\n+===========\n+\n+Profiling parallel code can be tricky. ``dask.diagnostics`` provides\n+functionality to aid in profiling and inspecting dask graph execution.\n+\n+\n+Scheduler Callbacks\n+-------------------\n+\n+Schedulers based on ``dask.async.get_async`` (currently\n+``dask.async.get_sync``, ``dask.threaded.get``, and\n+``dask.multiprocessing.get``) accept two callbacks, allowing for inspection of\n+dask execution. The callbacks are:\n+\n+1. ``start_callback(self, key, dask, state)``\n+    Callback ran every time a new task is started. Receives the key of the\n+    task to be run, the dask, and the scheduler state. The final callback\n+    receives ``None`` instead of a key, as no new tasks were added at that\n+    tick.\n+2. ``end_callback(self, key, dask, state, id)``\n+    Callback ran every time a task is finished. Receives the key of the\n+    task to be run, the dask, the scheduler state, and the id of the worker\n+    that ran the task. The final callback receives ``None`` none for both key\n+    worker id, as no new tasks were finished at that tick.\n+\n+\n+ Callbacks for common use cases are provided in ``dask.diagnostics``.\n+\n+\n+Profiler\n+--------\n+\n+The ``Profiler`` class profiles dask execution at the task level. To use,\n+create a profiler from a scheduler ``get`` function:\n+\n+.. code-block:: python\n+\n+    >>> from dask.array.core import get\n+    >>> from dask.diagnostics import Profiler\n+    >>> array_profiler = Profiler(get)\n+\n+The ``get`` method of the profiler then works like a normal scheduler, but\n+records the following information for each task during execution:\n+\n+1. Key\n+2. Task\n+3. Start time in seconds since the epoch\n+4. Finish time in seconds since the epoch\n+5. Worker id\n+\n+.. code-block:: python\n+\n+    >>> import dask.array as da\n+    >>> a = da.random.random(size=(1000,10000), chunks=(1000,1000))\n+    >>> q, r = da.linalg.qr(a)","path":"docs/source/diagnostics.rst","position":56,"original_position":56,"commit_id":"ac27053468f82d593ac5fc9c6e75eee3ce536a4e","original_commit_id":"ac27053468f82d593ac5fc9c6e75eee3ce536a4e","user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"body":"hmmm, this got fixed, but never committed apparently.","created_at":"2015-06-29T22:33:55Z","updated_at":"2015-06-29T22:33:55Z","html_url":"https://github.com/ContinuumIO/dask/pull/363#discussion_r33522073","pull_request_url":"https://api.github.com/repos/ContinuumIO/dask/pulls/363","_links":{"self":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/comments/33522073"},"html":{"href":"https://github.com/ContinuumIO/dask/pull/363#discussion_r33522073"},"pull_request":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/363"}}},"pull_request":{"url":"https://api.github.com/repos/ContinuumIO/dask/pulls/363","id":38488340,"html_url":"https://github.com/ContinuumIO/dask/pull/363","diff_url":"https://github.com/ContinuumIO/dask/pull/363.diff","patch_url":"https://github.com/ContinuumIO/dask/pull/363.patch","issue_url":"https://api.github.com/repos/ContinuumIO/dask/issues/363","number":363,"state":"open","locked":false,"title":"Profiler","user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"body":"`Profiler` takes a `get` function based on `get_async`, and a worker type. The `get` method of the resulting object then profiles dask execution at a task level. This records tuples of `(key, task, worker_id, start_time, end_time)`.\r\n\r\nBased on #356.","created_at":"2015-06-24T21:13:59Z","updated_at":"2015-06-29T22:33:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ddfd926789ad8bd156ec78f542b9afee2d6fcad8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ContinuumIO/dask/pulls/363/commits","review_comments_url":"https://api.github.com/repos/ContinuumIO/dask/pulls/363/comments","review_comment_url":"https://api.github.com/repos/ContinuumIO/dask/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ContinuumIO/dask/issues/363/comments","statuses_url":"https://api.github.com/repos/ContinuumIO/dask/statuses/ac27053468f82d593ac5fc9c6e75eee3ce536a4e","head":{"label":"jcrist:profile","ref":"profile","sha":"ac27053468f82d593ac5fc9c6e75eee3ce536a4e","user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"repo":{"id":32761805,"name":"dask","full_name":"jcrist/dask","owner":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jcrist/dask","description":"Minimal task scheduling abstraction","fork":true,"url":"https://api.github.com/repos/jcrist/dask","forks_url":"https://api.github.com/repos/jcrist/dask/forks","keys_url":"https://api.github.com/repos/jcrist/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jcrist/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jcrist/dask/teams","hooks_url":"https://api.github.com/repos/jcrist/dask/hooks","issue_events_url":"https://api.github.com/repos/jcrist/dask/issues/events{/number}","events_url":"https://api.github.com/repos/jcrist/dask/events","assignees_url":"https://api.github.com/repos/jcrist/dask/assignees{/user}","branches_url":"https://api.github.com/repos/jcrist/dask/branches{/branch}","tags_url":"https://api.github.com/repos/jcrist/dask/tags","blobs_url":"https://api.github.com/repos/jcrist/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jcrist/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jcrist/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/jcrist/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jcrist/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/jcrist/dask/languages","stargazers_url":"https://api.github.com/repos/jcrist/dask/stargazers","contributors_url":"https://api.github.com/repos/jcrist/dask/contributors","subscribers_url":"https://api.github.com/repos/jcrist/dask/subscribers","subscription_url":"https://api.github.com/repos/jcrist/dask/subscription","commits_url":"https://api.github.com/repos/jcrist/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/jcrist/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/jcrist/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/jcrist/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/jcrist/dask/contents/{+path}","compare_url":"https://api.github.com/repos/jcrist/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jcrist/dask/merges","archive_url":"https://api.github.com/repos/jcrist/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jcrist/dask/downloads","issues_url":"https://api.github.com/repos/jcrist/dask/issues{/number}","pulls_url":"https://api.github.com/repos/jcrist/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/jcrist/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/jcrist/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jcrist/dask/labels{/name}","releases_url":"https://api.github.com/repos/jcrist/dask/releases{/id}","created_at":"2015-03-23T22:11:35Z","updated_at":"2015-03-28T16:10:40Z","pushed_at":"2015-06-29T21:39:44Z","git_url":"git://github.com/jcrist/dask.git","ssh_url":"git@github.com:jcrist/dask.git","clone_url":"https://github.com/jcrist/dask.git","svn_url":"https://github.com/jcrist/dask","homepage":"http://dask.pydata.org","size":6053,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ContinuumIO:master","ref":"master","sha":"78a3a3a05e8905e7ecb833c685e8b6aa5db2aae2","user":{"login":"ContinuumIO","id":1158637,"avatar_url":"https://avatars.githubusercontent.com/u/1158637?v=3","gravatar_id":"","url":"https://api.github.com/users/ContinuumIO","html_url":"https://github.com/ContinuumIO","followers_url":"https://api.github.com/users/ContinuumIO/followers","following_url":"https://api.github.com/users/ContinuumIO/following{/other_user}","gists_url":"https://api.github.com/users/ContinuumIO/gists{/gist_id}","starred_url":"https://api.github.com/users/ContinuumIO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ContinuumIO/subscriptions","organizations_url":"https://api.github.com/users/ContinuumIO/orgs","repos_url":"https://api.github.com/users/ContinuumIO/repos","events_url":"https://api.github.com/users/ContinuumIO/events{/privacy}","received_events_url":"https://api.github.com/users/ContinuumIO/received_events","type":"Organization","site_admin":false},"repo":{"id":28782747,"name":"dask","full_name":"ContinuumIO/dask","owner":{"login":"ContinuumIO","id":1158637,"avatar_url":"https://avatars.githubusercontent.com/u/1158637?v=3","gravatar_id":"","url":"https://api.github.com/users/ContinuumIO","html_url":"https://github.com/ContinuumIO","followers_url":"https://api.github.com/users/ContinuumIO/followers","following_url":"https://api.github.com/users/ContinuumIO/following{/other_user}","gists_url":"https://api.github.com/users/ContinuumIO/gists{/gist_id}","starred_url":"https://api.github.com/users/ContinuumIO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ContinuumIO/subscriptions","organizations_url":"https://api.github.com/users/ContinuumIO/orgs","repos_url":"https://api.github.com/users/ContinuumIO/repos","events_url":"https://api.github.com/users/ContinuumIO/events{/privacy}","received_events_url":"https://api.github.com/users/ContinuumIO/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ContinuumIO/dask","description":"Task scheduling and blocked algorithms for parallel processing","fork":false,"url":"https://api.github.com/repos/ContinuumIO/dask","forks_url":"https://api.github.com/repos/ContinuumIO/dask/forks","keys_url":"https://api.github.com/repos/ContinuumIO/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ContinuumIO/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ContinuumIO/dask/teams","hooks_url":"https://api.github.com/repos/ContinuumIO/dask/hooks","issue_events_url":"https://api.github.com/repos/ContinuumIO/dask/issues/events{/number}","events_url":"https://api.github.com/repos/ContinuumIO/dask/events","assignees_url":"https://api.github.com/repos/ContinuumIO/dask/assignees{/user}","branches_url":"https://api.github.com/repos/ContinuumIO/dask/branches{/branch}","tags_url":"https://api.github.com/repos/ContinuumIO/dask/tags","blobs_url":"https://api.github.com/repos/ContinuumIO/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ContinuumIO/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ContinuumIO/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/ContinuumIO/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ContinuumIO/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/ContinuumIO/dask/languages","stargazers_url":"https://api.github.com/repos/ContinuumIO/dask/stargazers","contributors_url":"https://api.github.com/repos/ContinuumIO/dask/contributors","subscribers_url":"https://api.github.com/repos/ContinuumIO/dask/subscribers","subscription_url":"https://api.github.com/repos/ContinuumIO/dask/subscription","commits_url":"https://api.github.com/repos/ContinuumIO/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/ContinuumIO/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/ContinuumIO/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/ContinuumIO/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/ContinuumIO/dask/contents/{+path}","compare_url":"https://api.github.com/repos/ContinuumIO/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ContinuumIO/dask/merges","archive_url":"https://api.github.com/repos/ContinuumIO/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ContinuumIO/dask/downloads","issues_url":"https://api.github.com/repos/ContinuumIO/dask/issues{/number}","pulls_url":"https://api.github.com/repos/ContinuumIO/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/ContinuumIO/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/ContinuumIO/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ContinuumIO/dask/labels{/name}","releases_url":"https://api.github.com/repos/ContinuumIO/dask/releases{/id}","created_at":"2015-01-04T18:50:00Z","updated_at":"2015-06-29T18:36:38Z","pushed_at":"2015-06-29T22:28:57Z","git_url":"git://github.com/ContinuumIO/dask.git","ssh_url":"git@github.com:ContinuumIO/dask.git","clone_url":"https://github.com/ContinuumIO/dask.git","svn_url":"https://github.com/ContinuumIO/dask","homepage":"http://dask.pydata.org","size":9175,"stargazers_count":98,"watchers_count":98,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":58,"forks":24,"open_issues":58,"watchers":98,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/363"},"html":{"href":"https://github.com/ContinuumIO/dask/pull/363"},"issue":{"href":"https://api.github.com/repos/ContinuumIO/dask/issues/363"},"comments":{"href":"https://api.github.com/repos/ContinuumIO/dask/issues/363/comments"},"review_comments":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/363/comments"},"review_comment":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ContinuumIO/dask/pulls/363/commits"},"statuses":{"href":"https://api.github.com/repos/ContinuumIO/dask/statuses/ac27053468f82d593ac5fc9c6e75eee3ce536a4e"}}}}