{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/33558197","id":33558197,"diff_hunk":"@@ -497,18 +501,29 @@ func (r *Range) SetLastVerificationTimestamp(timestamp proto.Timestamp) error {\n func (r *Range) AddCmd(ctx context.Context, call proto.Call, wait bool) error {\n \targs, reply := call.Args, call.Reply\n \theader := args.Header()\n+\t// TODO Most requests are ok (those from store.ExecuteCmd) but others get here","path":"storage/range.go","position":29,"original_position":29,"commit_id":"2dbc43dd9d275f6de5b5f1d4fe0df85ee14b4564","original_commit_id":"2dbc43dd9d275f6de5b5f1d4fe0df85ee14b4564","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"body":"done","created_at":"2015-06-30T10:09:54Z","updated_at":"2015-06-30T10:09:54Z","html_url":"https://github.com/cockroachdb/cockroach/pull/1446#discussion_r33558197","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/33558197"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/1446#discussion_r33558197"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446","id":38334674,"html_url":"https://github.com/cockroachdb/cockroach/pull/1446","diff_url":"https://github.com/cockroachdb/cockroach/pull/1446.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/1446.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1446","number":1446,"state":"open","locked":false,"title":"Add basic request tracing facility","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"body":"For discussion:\r\nadded the `tracer` package, providing facilities to trace requests through the cluster.\r\n\r\nreworked API based on the suggestions from the first proposal: a Trace is\r\nautonomous, and submits itself on `Finalize()`. For tracing requests, have\r\n\r\n- `trace.Event(\"x happened\")` and\r\n- `fn := trace.Epoch(\"stuff begins now\"); fn() // stuff ends now`, with its<br />\r\n  most convenient use case `defer trace.Epoch(\"x\")()`.\r\n\r\nEpochs and Events generate the same entries, but the former with nonzero\r\nduration. Each entry has a depth, so nested epochs can be displayed nicely:\r\n\r\nsingle-request increment, node side. Note that the total duration of the call\r\n(609.166us) is right at the top; the nested epochs below divvy it up.\r\n```\r\nID                 Origin              Ts                 Dur          Desc                    File\r\n1435427552[...]    macts.local:8001    13:52:32.200487    609.166µs    node                    server/node.go:564\r\n1435427552[...]    macts.local:8001    13:52:32.200515    576.619µs    ·executing Increment    storage/store.go:1271\r\n1435427552[...]    macts.local:8001    13:52:32.200554    538.349µs    ··range                 storage/range.go:517\r\n1435427552[...]    macts.local:8001    13:52:32.200561    0            ···write path           storage/range.go:526\r\n1435427552[...]    macts.local:8001    13:52:32.200576    515.994µs    ···raft                 storage/range.go:703\r\n1435427552[...]    macts.local:8001    13:52:32.200583    48.923µs     ····proposing           storage/range.go:765\r\n1435427552[...]    macts.local:8001    13:52:32.200971    107.385µs    ····executing           storage/range.go:799\r\n```\r\n\r\n`EndTransaction` on the client gateway side:\r\n```\r\nI0627 13:52:32.785017   53305 monitor.go:211] received trace:\r\nID          Origin              Ts                 Dur            Desc                             File\r\n2f2ecd58    macts.local:8001    13:52:32.743443    11.079554ms    coordinated sender               kv/txn_coord_sender.go:324\r\n2f2ecd58    macts.local:8001    13:52:32.743451    1.57359ms      ·sending EndTransaction          kv/dist_sender.go:638\r\n2f2ecd58    macts.local:8001    13:52:32.743455    5.607µs        ··meta descriptor lookup         kv/dist_sender.go:675\r\n2f2ecd58    macts.local:8001    13:52:32.743483    0              ··sending to macts.local:8001    rpc/send.go:178\r\n```\r\n\r\nTransactional `Get` which hits an intent, attempts to resolve it\r\nasynchronously, backs off and gets an error not retryable at the store level.\r\n```\r\nID          Origin              Ts                 Dur             Desc                   File\r\n3de447e3    macts.local:8001    13:52:32.888115    139.456068ms    node                   server/node.go:564\r\n3de447e3    macts.local:8001    13:52:32.888125    126.494154ms    ·executing Get         storage/store.go:1271\r\n3de447e3    macts.local:8001    13:52:32.888128    110.367783ms    ··range                storage/range.go:517\r\n3de447e3    macts.local:8001    13:52:32.888131    110.364629ms    ···read path           storage/range.go:523\r\n3de447e3    macts.local:8001    13:52:32.998509    16.098773ms     ··intent resolution    storage/store.go:1343\r\n3de447e3    macts.local:8001    13:52:33.014457    150.888µs       ···range               storage/range.go:517\r\n3de447e3    macts.local:8001    13:52:33.014460    0               ····write path         storage/range.go:526\r\n3de447e3    macts.local:8001    13:52:33.014489    0               ····raft [async]       storage/range.go:698\r\n3de447e3    macts.local:8001    13:52:33.014625    165.802µs       ·backoff               storage/store.go:1264\r\n3de447e3    macts.local:8001    13:52:33.014796    12.765902ms     ·executing Get         storage/store.go:1271\r\n3de447e3    macts.local:8001    13:52:33.014802    12.759787ms     ··range                storage/range.go:517\r\n3de447e3    macts.local:8001    13:52:33.014805    12.755809ms     ···read path           storage/range.go:523\r\n3de447e3    macts.local:8001    13:52:33.027566    0               ·not successful        storage/store.go:1266\r\n```\r\n\r\nTODO:\r\n\r\n- convert to protos\r\n- metadata handling? (ex: peer at request boundary)\r\n- figure out persisting and viewing Traces (as discussed in PR)","created_at":"2015-06-23T13:14:05Z","updated_at":"2015-06-30T10:09:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5cacc2f457f8aee696b9006e64529049e3879eb6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1446/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/2dbc43dd9d275f6de5b5f1d4fe0df85ee14b4564","head":{"label":"tschottdorf:tracer","ref":"tracer","sha":"2dbc43dd9d275f6de5b5f1d4fe0df85ee14b4564","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"repo":{"id":23832429,"name":"cockroach","full_name":"tschottdorf/cockroach","owner":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tschottdorf/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":true,"url":"https://api.github.com/repos/tschottdorf/cockroach","forks_url":"https://api.github.com/repos/tschottdorf/cockroach/forks","keys_url":"https://api.github.com/repos/tschottdorf/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tschottdorf/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tschottdorf/cockroach/teams","hooks_url":"https://api.github.com/repos/tschottdorf/cockroach/hooks","issue_events_url":"https://api.github.com/repos/tschottdorf/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/tschottdorf/cockroach/events","assignees_url":"https://api.github.com/repos/tschottdorf/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/tschottdorf/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/tschottdorf/cockroach/tags","blobs_url":"https://api.github.com/repos/tschottdorf/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tschottdorf/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tschottdorf/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/tschottdorf/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tschottdorf/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/tschottdorf/cockroach/languages","stargazers_url":"https://api.github.com/repos/tschottdorf/cockroach/stargazers","contributors_url":"https://api.github.com/repos/tschottdorf/cockroach/contributors","subscribers_url":"https://api.github.com/repos/tschottdorf/cockroach/subscribers","subscription_url":"https://api.github.com/repos/tschottdorf/cockroach/subscription","commits_url":"https://api.github.com/repos/tschottdorf/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/tschottdorf/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/tschottdorf/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/tschottdorf/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/tschottdorf/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/tschottdorf/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tschottdorf/cockroach/merges","archive_url":"https://api.github.com/repos/tschottdorf/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tschottdorf/cockroach/downloads","issues_url":"https://api.github.com/repos/tschottdorf/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/tschottdorf/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/tschottdorf/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/tschottdorf/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tschottdorf/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/tschottdorf/cockroach/releases{/id}","created_at":"2014-09-09T12:26:29Z","updated_at":"2015-04-30T01:27:18Z","pushed_at":"2015-06-30T10:07:42Z","git_url":"git://github.com/tschottdorf/cockroach.git","ssh_url":"git@github.com:tschottdorf/cockroach.git","clone_url":"https://github.com/tschottdorf/cockroach.git","svn_url":"https://github.com/tschottdorf/cockroach","homepage":null,"size":25132,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":16,"forks":0,"open_issues":16,"watchers":0,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"09e8404e00b826e4f15e33c69e65e853adba4901","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","created_at":"2014-02-06T00:18:47Z","updated_at":"2015-06-30T06:11:46Z","pushed_at":"2015-06-30T10:07:42Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":null,"size":42106,"stargazers_count":4813,"watchers_count":4813,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":352,"mirror_url":null,"open_issues_count":103,"forks":352,"open_issues":103,"watchers":4813,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/1446"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/1446"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/1446/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/1446/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/2dbc43dd9d275f6de5b5f1d4fe0df85ee14b4564"}}}}