{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1483","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1483/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1483/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1483/events","html_url":"https://github.com/cockroachdb/cockroach/issues/1483","id":90761390,"number":1483,"title":"Panic while running the bank app on resolveWriteIntentError due to nil pointer dereference","user":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"milestone":null,"comments":13,"created_at":"2015-06-24T19:12:06Z","updated_at":"2015-06-25T20:03:15Z","closed_at":null,"body":"[full stack trace](https://gist.githubusercontent.com/BramGruneir/a574350ef1c49cdcc37d/raw/gistfile1.txt)\r\n```\r\nI0624 14:58:11.773672   22542 store.go:1356] on Get: txn \"bank.go:154\" id=d9de0973 key=\"000000005\" pri=99.96220432 iso=SERIALIZABLE stat=PENDING epo=3 ts=1435172276.927472353,3 orig=1435172276.927472353,3 max=1435172274.682641506,0 failed to push \"bank.go:154\" id=2e92ce96 key=\"000000008\" pri=99.96220437 iso=SERIALIZABLE stat=PENDING epo=3 ts=1435172276.927472353,2 orig=1435172274.215435743,1 max=1435172274.215435743,1\r\nI0624 14:58:11.778768   22542 store.go:1356] on Get: txn \"bank.go:154\" id=aeccb05a key=\"000000006\" pri=99.96220432 iso=SERIALIZABLE stat=PENDING epo=5 ts=1435172276.927472353,2 orig=1435172276.927472353,2 max=1435172276.927472353,2 failed to push \"bank.go:154\" id=2e92ce96 key=\"000000008\" pri=99.96220437 iso=SERIALIZABLE stat=PENDING epo=3 ts=1435172276.927472353,2 orig=1435172274.215435743,1 max=1435172274.215435743,1\r\npanic: runtime error: invalid memory address or nil pointer dereference\r\n[signal 0xb code=0x1 addr=0x0 pc=0x4652a65]\r\n\r\ngoroutine 473487 [running]:\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).resolveWriteIntentError(0xc20815e000, 0x5439b80, 0xc20986cf90, 0xc20cd2ee80, 0xc208160090, 0x5447b68, 0xc208dfc100, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1385 +0x1055\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc20ac21570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1262 +0x2f3\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc2088060f0, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc20ac217b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20986cf90, 0x5447b68, 0xc208dfc100, 0x5447bc0, 0xc2090ea070, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20986cf30, 0x5447b68, 0xc208dfc100, 0x5447bc0, 0xc2090ea070, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc208dfc100, 0x5447bc0, 0xc2090ea070, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc208dfc100, 0xc2090ea070, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc20ac21f28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc20ac21f28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc20c347680, 0x4ae75e0, 0xc208dfc100, 0x16, 0x4add300, 0xc2090ea070, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 1 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/server/cli.runStart(0x4f32160, 0xc208040cd0, 0x0, 0x5)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/cli/start.go:140 +0xdcc\r\ngithub.com/spf13/cobra.(*Command).execute(0x4f32160, 0xc208040c80, 0x5, 0x5, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/spf13/cobra/command.go:495 +0x65c\r\ngithub.com/spf13/cobra.(*Command).Execute(0x4f30420, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/spf13/cobra/command.go:560 +0x18d\r\ngithub.com/cockroachdb/cockroach/server/cli.Run(0xc20800a010, 0x6, 0x6, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/cli/cli.go:118 +0x88\r\nmain.main()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/main.go:48 +0x29d\r\n\r\ngoroutine 5 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x4f704e0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1034 +0x75\r\ncreated by github.com/cockroachdb/cockroach/util/log.init·1\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:604 +0xcc\r\n\r\ngoroutine 17 [syscall, 5 minutes, locked to thread]:\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1\r\n\r\ngoroutine 7 [syscall, 5 minutes]:\r\nos/signal.loop()\r\n\t/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f\r\ncreated by os/signal.init·1\r\n\t/usr/local/go/src/os/signal/signal_unix.go:27 +0x35\r\n\r\ngoroutine 82 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/storage.func·013()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/queue.go:207 +0x2cb\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 10 [select]:\r\ngithub.com/cockroachdb/cockroach/rpc.(*RemoteClockMonitor).MonitorRemoteOffsets(0xc20801d680, 0xc208042840)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/clock_offset.go:158 +0x637\r\ngithub.com/cockroachdb/cockroach/server.func·010()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/server.go:106 +0x3a\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 11 [select]:\r\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc20807c510)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:261 +0x8f9\r\ngithub.com/cockroachdb/cockroach/kv.*TxnCoordSender.(github.com/cockroachdb/cockroach/kv.startStats)·fm()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:248 +0x27\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 12 [select]:\r\ngithub.com/cockroachdb/cockroach/gossip.func·021()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/gossip/server.go:156 +0x139\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 13 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/gossip.func·004()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:431 +0x1f2\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 14 [select]:\r\ngithub.com/cockroachdb/cockroach/gossip.func·005()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:455 +0x1ab\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 476316 [select]:\r\ngithub.com/cockroachdb/cockroach/rpc.sendOne(0xc2081f6000, 0x12a05f200, 0xc20c5634c0, 0x14, 0x4af1360, 0xc20b860780, 0x4adda00, 0xc20ad92d90, 0xc20c7a8de0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/send.go:235 +0x9af\r\ncreated by github.com/cockroachdb/cockroach/rpc.Send\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/send.go:173 +0x1578\r\n\r\ngoroutine 16 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/storage.ProcessStoreEvents(0x5439268, 0xc2080fc380, 0xc20810c290)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/feed.go:256 +0x61\r\ncreated by github.com/cockroachdb/cockroach/server/status.(*NodeStatusMonitor).StartMonitorFeed\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/status/monitor.go:106 +0x7f\r\n\r\ngoroutine 18 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/server/status.ProcessNodeEvents(0x54392d8, 0xc2080fc380, 0xc20810c2a0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/status/feed.go:84 +0x61\r\ncreated by github.com/cockroachdb/cockroach/server/status.(*NodeStatusMonitor).StartMonitorFeed\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/status/monitor.go:107 +0xe2\r\n\r\ngoroutine 476422 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c4a7640)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc20c65b600, 0xc208042701, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc20c2d3260, 0x5447b68, 0xc20c65b600, 0x5447bc0, 0xc2081fec40, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:599 +0x624\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc20c2d3260, 0x5447b68, 0xc20c65b600, 0x5447bc0, 0xc2081fec40, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0x6, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20ad20300, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc20a4ef7b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20c2d3260, 0x5447b68, 0xc20c65b600, 0x5447bc0, 0xc2081fec40, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20c2d3200, 0x5447b68, 0xc20c65b600, 0x5447bc0, 0xc2081fec40, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc20c65b600, 0x5447bc0, 0xc2081fec40, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc20c65b600, 0xc2081fec40, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc20a4eff28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc20a4eff28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc208cf6bc0, 0x4ae75e0, 0xc20c65b600, 0x16, 0x4add300, 0xc2081fec40, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 35 [runnable, locked to thread]:\r\ngithub.com/cockroachdb/cockroach/storage/engine._Cfunc_DBIterDestroy(0x6200000)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/engine/:93 +0x45\r\ngithub.com/cockroachdb/cockroach/storage/engine.(*rocksDBIterator).Close(0xc2081bc490)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:711 +0x2b\r\ngithub.com/cockroachdb/cockroach/storage/engine.MVCCIterate(0x5437e68, 0xc208042720, 0xc20c369560, 0x12, 0x12, 0xc20c3695a0, 0x12, 0x12, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:1028 +0x592\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).Entries(0xc208160090, 0x15c19, 0x15c1a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range_raftstorage.go:95 +0x29b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).Term(0xc208160090, 0x15c19, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range_raftstorage.go:112 +0x6c\r\ngithub.com/coreos/etcd/raft.(*raftLog).term(0xc20819a960, 0x15c19, 0x4f798b0)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/log.go:209 +0xcf\r\ngithub.com/coreos/etcd/raft.(*raftLog).matchTerm(0xc20819a960, 0x15c19, 0xa, 0xc208cb9300)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/log.go:239 +0x32\r\ngithub.com/coreos/etcd/raft.(*raftLog).maybeAppend(0xc20819a960, 0x15c19, 0xa, 0x15c18, 0x0, 0x0, 0x0, 0x15c19, 0x80)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/log.go:74 +0x58\r\ngithub.com/coreos/etcd/raft.(*raft).handleAppendEntries(0xc2080ee600, 0x3, 0x100000001, 0x200000002, 0xa, 0xa, 0x15c19, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:657 +0x24f\r\ngithub.com/coreos/etcd/raft.stepFollower(0xc2080ee600, 0x3, 0x100000001, 0x200000002, 0xa, 0xa, 0x15c19, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:628 +0x3ba\r\ngithub.com/coreos/etcd/raft.(*raft).Step(0xc2080ee600, 0x3, 0x100000001, 0x200000002, 0xa, 0xa, 0x15c19, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:487 +0x259\r\ngithub.com/coreos/etcd/raft.(*multiNode).run(0xc2081046c0)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:236 +0xd0b\r\ncreated by github.com/coreos/etcd/raft.StartMultiNode\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:56 +0xb6\r\n\r\ngoroutine 36 [select]:\r\ngithub.com/coreos/etcd/raft.(*multiNode).step(0xc2081046c0, 0x54474a0, 0xc208028b58, 0x1, 0x3, 0x100000001, 0x200000002, 0xa, 0xa, 0x15c19, ...)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:407 +0x1ee\r\ngithub.com/coreos/etcd/raft.(*multiNode).Step(0xc2081046c0, 0x54474a0, 0xc208028b58, 0x1, 0x3, 0x100000001, 0x200000002, 0xa, 0xa, 0x15c19, ...)\r\n\t/Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:438 +0x135\r\ngithub.com/cockroachdb/cockroach/multiraft.func·005()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:521 +0xba5\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 37 [semacquire]:\r\nsync.(*Mutex).Lock(0xc2081600e8)\r\n\t/usr/local/go/src/sync/mutex.go:66 +0xd3\r\nsync.(*RWMutex).Lock(0xc2081600e8)\r\n\t/usr/local/go/src/sync/rwmutex.go:82 +0x33\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).processRaftCommand(0xc208160090, 0xc20b95fff0, 0x10, 0x15c17, 0x1, 0x100000001, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:756 +0xd7\r\ngithub.com/cockroachdb/cockroach/storage.func·043()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1492 +0xa6b\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 20 [select]:\r\ngithub.com/cockroachdb/cockroach/server.func·006()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:451 +0x78e\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 224 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc20806d480, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20806d480, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc20806d420, 0xc2081c0000, 0x1000, 0x1000, 0x0, 0x5437028, 0xc208ef4660)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc20819e280, 0xc2081c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nbufio.(*Reader).fill(0xc2081c93e0)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadByte(0xc2081c93e0, 0xc20c288700, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:196 +0x86\r\nencoding/binary.ReadUvarint(0x5446c00, 0xc2081c93e0, 0x5432da8, 0x0, 0x0)\r\n\t/usr/local/go/src/encoding/binary/varint.go:110 +0x5b\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc2082041c0, 0x5446bc8, 0xc208204330, 0x0, 0x4cb4850, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:91 +0x76\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc2082041c0, 0xc2081c93e0, 0xc208204330, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:180 +0x86\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc2082041c0, 0xc20963cc20, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:64 +0x6e\r\nnet/rpc.(*Server).readRequestHeader(0xc2080fc200, 0x5446b88, 0xc2082041c0, 0x0, 0x0, 0xc20963cc20, 0xc20821dd00, 0x0, 0x0)\r\n\t/usr/local/go/src/net/rpc/server.go:575 +0x97\r\nnet/rpc.(*Server).readRequest(0xc2080fc200, 0x5446b88, 0xc2082041c0, 0xc208061300, 0xc2095ef760, 0x4aa8100, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:542 +0xa2\r\nnet/rpc.(*Server).ServeCodec(0xc2080fc200, 0x5446b88, 0xc2082041c0)\r\n\t/usr/local/go/src/net/rpc/server.go:461 +0x6d\r\ngithub.com/cockroachdb/cockroach/rpc.(*Server).serveConn(0xc208061280, 0x5442920, 0xc20819e280, 0xc2082175a0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:254 +0x95\r\ngithub.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc208061280, 0x5446ac8, 0xc2081c2500, 0xc208168f70)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:115 +0x47d\r\nnet/http.serverHandler.ServeHTTP(0xc2080426c0, 0x5446ac8, 0xc2081c2500, 0xc208168f70)\r\n\t/usr/local/go/src/net/http/server.go:1703 +0x19a\r\nnet/http.(*conn).serve(0xc2081c2460)\r\n\t/usr/local/go/src/net/http/server.go:1204 +0xb57\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n\r\ngoroutine 40 [select]:\r\ngithub.com/cockroachdb/cockroach/storage.func·036()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:584 +0x39e\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 41 [select]:\r\ngithub.com/cockroachdb/cockroach/storage.func·037()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:604 +0x39e\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 476407 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c4e1780)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc20b7be280, 0xc209324d00, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addWriteCmd(0xc208160090, 0x5439b80, 0xc20c31e810, 0x545a638, 0xc20b7be280, 0x545a730, 0xc208307d50, 0xc209357501, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:640 +0x14d\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc20c31e810, 0x545a638, 0xc20b7be280, 0x545a730, 0xc208307d50, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:512 +0x4f6\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0x6, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20c4e1760, 0x16, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc2093257b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20c31e810, 0x545a638, 0xc20b7be280, 0x545a730, 0xc208307d50, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20c31e7b0, 0x545a638, 0xc20b7be280, 0x545a730, 0xc208307d50, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x545a638, 0xc20b7be280, 0x545a730, 0xc208307d50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).InternalPushTxn(0xc20806e500, 0xc20b7be280, 0xc208307d50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:617 +0xb1\r\nreflect.Value.call(0x49d7640, 0x4aebbe0, 0x13, 0x4b16390, 0x4, 0xc209325f28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7640, 0x4aebbe0, 0x13, 0xc209325f28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc20818c568, 0xc208061b00, 0xc208252380, 0x4af1360, 0xc20b7be280, 0x16, 0x4adda00, 0xc208307d50, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 50 [select]:\r\ngithub.com/cockroachdb/cockroach/multiraft.func·008()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:145 +0x90a\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 83 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/storage.func·013()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/queue.go:207 +0x2cb\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 84 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/storage.func·013()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/queue.go:207 +0x2cb\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 85 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/storage.func·013()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/queue.go:207 +0x2cb\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 86 [select, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/storage.func·013()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/queue.go:207 +0x2cb\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 87 [select]:\r\ngithub.com/cockroachdb/cockroach/storage.(*rangeScanner).waitAndProcess(0xc2081022a0, 0xecd1cf4c2, 0xe21bbfa30, 0x4f6ff00, 0xc2080fc1c0, 0xc208042840, 0xc20c2935f0, 0xc208160090, 0xc200000001)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:192 +0x6f5\r\ngithub.com/cockroachdb/cockroach/storage.func·031(0xc208160090, 0xc20814e150)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:240 +0x71\r\ngithub.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc20814e1b0, 0xc20b06cf00)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:233 +0x280\r\ngithub.com/cockroachdb/cockroach/storage.func·032()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:242 +0x5a5\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 473893 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c681840)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc2088ee100, 0xdeaddeaddeadde01, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc20c8d5950, 0x5447b68, 0xc2088ee100, 0x5447bc0, 0xc20b65ad20, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:599 +0x624\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc20c8d5950, 0x5447b68, 0xc2088ee100, 0x5447bc0, 0xc20b65ad20, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc20c5fd570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc208435f40, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc20c5fd7b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20c8d5950, 0x5447b68, 0xc2088ee100, 0x5447bc0, 0xc20b65ad20, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20c8d58f0, 0x5447b68, 0xc2088ee100, 0x5447bc0, 0xc20b65ad20, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc2088ee100, 0x5447bc0, 0xc20b65ad20, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc2088ee100, 0xc20b65ad20, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc20c5fdf28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc20c5fdf28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc2082821a0, 0x4ae75e0, 0xc2088ee100, 0x16, 0x4add300, 0xc20b65ad20, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 98 [select]:\r\ngithub.com/cockroachdb/cockroach/rpc.(*Client).runHeartbeat(0xc2081f6000, 0xc208042840)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/client.go:243 +0x367\r\ngithub.com/cockroachdb/cockroach/rpc.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/client.go:125 +0x2a7\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 114 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc20806c8b0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20806c8b0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc20806c850, 0xc208202000, 0x1000, 0x1000, 0x0, 0x5437028, 0xc208ef4040)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc2081bc080, 0xc208202000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nbufio.(*Reader).fill(0xc20813c180)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadByte(0xc20813c180, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:196 +0x86\r\nencoding/binary.ReadUvarint(0x5446c00, 0xc20813c180, 0x42bd7e9, 0x0, 0x0)\r\n\t/usr/local/go/src/encoding/binary/varint.go:110 +0x5b\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc208204000, 0x5446bc8, 0xc208204170, 0x0, 0x4cb4850, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:91 +0x76\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc208204000, 0xc20813c180, 0xc208204170, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:180 +0x86\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc208204000, 0xc20c05c020, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:64 +0x6e\r\nnet/rpc.(*Server).readRequestHeader(0xc2080fc200, 0x5446b88, 0xc208204000, 0x0, 0x0, 0xc20c05c020, 0xc20817a900, 0x0, 0x0)\r\n\t/usr/local/go/src/net/rpc/server.go:575 +0x97\r\nnet/rpc.(*Server).readRequest(0xc2080fc200, 0x5446b88, 0xc208204000, 0xc208061b00, 0xc20955b780, 0x4af1360, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:542 +0xa2\r\nnet/rpc.(*Server).ServeCodec(0xc2080fc200, 0x5446b88, 0xc208204000)\r\n\t/usr/local/go/src/net/rpc/server.go:461 +0x6d\r\ngithub.com/cockroachdb/cockroach/rpc.(*Server).serveConn(0xc208061280, 0x5442920, 0xc2081bc080, 0xc20818a560)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:254 +0x95\r\ngithub.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc208061280, 0x5446ac8, 0xc2081ae1e0, 0xc2080732b0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:115 +0x47d\r\nnet/http.serverHandler.ServeHTTP(0xc2080426c0, 0x5446ac8, 0xc2081ae1e0, 0xc2080732b0)\r\n\t/usr/local/go/src/net/http/server.go:1703 +0x19a\r\nnet/http.(*conn).serve(0xc2081c21e0)\r\n\t/usr/local/go/src/net/http/server.go:1204 +0xb57\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n\r\ngoroutine 476458 [semacquire]:\r\nsync.(*Mutex).Lock(0xc2081600e0)\r\n\t/usr/local/go/src/sync/mutex.go:66 +0xd3\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).redirectOnOrAcquireLeaderLease(0xc208160090, 0x13eac19920ebd504, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:371 +0x5e\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc208cb2060, 0x5453e48, 0xc20c323d40, 0x5453ea0, 0xc20ae56100, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:602 +0x688\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc208cb2060, 0x5453e48, 0xc20c323d40, 0x5453ea0, 0xc20ae56100, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0x6, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20ad20a50, 0xb, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc208db57b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc208cb2060, 0x5453e48, 0xc20c323d40, 0x5453ea0, 0xc20ae56100, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc208cb2000, 0x5453e48, 0xc20c323d40, 0x5453ea0, 0xc20ae56100, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5453e48, 0xc20c323d40, 0x5453ea0, 0xc20ae56100, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Scan(0xc20806e500, 0xc20c323d40, 0xc20ae56100, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:589 +0xb1\r\nreflect.Value.call(0x49d7a00, 0x4aebd00, 0x13, 0x4b16390, 0x4, 0xc208db5f28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7a00, 0x4aebd00, 0x13, 0xc208db5f28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061e00, 0xc208114b80, 0x4af0080, 0xc20c323d40, 0x16, 0x4ae2840, 0xc20ae56100, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 21 [select]:\r\ngithub.com/cockroachdb/cockroach/server.func·007()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:522 +0x22c\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 22 [select]:\r\ngithub.com/cockroachdb/cockroach/server.func·003()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:416 +0x15a\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 23 [select]:\r\ngithub.com/cockroachdb/cockroach/ts.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/ts/db.go:77 +0x141\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 24 [select]:\r\ngithub.com/cockroachdb/cockroach/ts.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/ts/db.go:77 +0x141\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 25 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc20806cfb0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20806cfb0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).accept(0xc20806cf50, 0x0, 0x5437028, 0xc20ad20280)\r\n\t/usr/local/go/src/net/fd_unix.go:419 +0x40b\r\nnet.(*TCPListener).AcceptTCP(0xc2080381a0, 0xc208089e20, 0x0, 0x0)\r\n\t/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e\r\nnet.(*TCPListener).Accept(0xc2080381a0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c\r\nnet/http.(*Server).Serve(0xc2080426c0, 0x54381f8, 0xc2080381a0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1728 +0x92\r\ngithub.com/cockroachdb/cockroach/rpc.func·004()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:165 +0x53\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 26 [chan receive, 4 minutes]:\r\ngithub.com/cockroachdb/cockroach/rpc.func·005()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/server.go:173 +0x55\r\ngithub.com/cockroachdb/cockroach/util.func·001()\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51\r\ncreated by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3\r\n\r\ngoroutine 472335 [select]:\r\ngithub.com/cockroachdb/cockroach/rpc.Send(0x1, 0x1, 0x1dcd6500, 0x12a05f200, 0xc20c54dc40, 0x14, 0xc20aa3c770, 0x1, 0x1, 0xc20c54dbe0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/send.go:176 +0x12c0\r\ngithub.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc208061380, 0x1, 0xc20c25d550, 0x1, 0x1, 0x1, 0x545a638, 0xc20aca8780, 0x545a730, 0xc20b5fc150, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:465 +0x8ae\r\ngithub.com/cockroachdb/cockroach/kv.(*DistSender).sendAttempt(0xc208061380, 0xc2081032d0, 0x545a638, 0xc20aca8780, 0x545a730, 0xc20b5fc150, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:506 +0x23c\r\ngithub.com/cockroachdb/cockroach/kv.func·008(0xc200000000, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:681 +0x359\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20c54db40, 0x1b, 0xee6b280, 0x6fc23ac00, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc20b479d48, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc208061380, 0x5439b40, 0xc208028bf0, 0x545a638, 0xc20aca8780, 0x545a730, 0xc20b5fc150, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:682 +0x6b9\r\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendOne(0xc20807c510, 0x545a638, 0xc20aca8780, 0x545a730, 0xc20b5fc150, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:390 +0x273\r\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendBatch(0xc20807c510, 0xc20b34c900, 0xc20afdf400)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:535 +0x753\r\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc20807c510, 0x5439b40, 0xc208028bf0, 0x545a690, 0xc20b34c900, 0x545a6e8, 0xc20afdf400, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:315 +0x134\r\ngithub.com/cockroachdb/cockroach/client.(*DB).send(0xc20806cee0, 0xc20b8e3540, 0x1, 0x1, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/client/db.go:432 +0x3e2\r\ngithub.com/cockroachdb/cockroach/client.(*DB).Run(0xc20806cee0, 0xc20b364c00, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/client/db.go:390 +0x8e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).resolveWriteIntentError(0xc20815e000, 0x5439b80, 0xc20c366bd0, 0xc20b8e3280, 0xc208160090, 0x5447b68, 0xc20b7a8800, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1354 +0x750\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc20b47b570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1262 +0x2f3\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20c364eb0, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc20b47b7b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20c366bd0, 0x5447b68, 0xc20b7a8800, 0x5447bc0, 0xc208f67d50, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20c366b70, 0x5447b68, 0xc20b7a8800, 0x5447bc0, 0xc208f67d50, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc20b7a8800, 0x5447bc0, 0xc208f67d50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc20b7a8800, 0xc208f67d50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc20b47bf28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc20b47bf28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc20a53eb60, 0x4ae75e0, 0xc20b7a8800, 0x16, 0x4add300, 0xc208f67d50, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 27 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc20806c840, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20806c840, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc20806c7e0, 0xc208206000, 0x1000, 0x1000, 0x0, 0x5437028, 0xc20ad21300)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc20819e080, 0xc208206000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nbufio.(*Reader).fill(0xc208042900)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadByte(0xc208042900, 0xff4, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:196 +0x86\r\nencoding/binary.ReadUvarint(0x5446c00, 0xc208042900, 0x4f70f20, 0x0, 0x0)\r\n\t/usr/local/go/src/encoding/binary/varint.go:110 +0x5b\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc20819c680, 0x5446cd0, 0xc20819c7d0, 0x0, 0x4cb4850, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:91 +0x76\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc20819c680, 0xc20819c7d0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x86\r\ngithub.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc20819c680, 0xc20814e960, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x54\r\nnet/rpc.(*Client).input(0xc208042960)\r\n\t/usr/local/go/src/net/rpc/client.go:109 +0xb6\r\ncreated by net/rpc.NewClientWithCodec\r\n\t/usr/local/go/src/net/rpc/client.go:201 +0xd0\r\n\r\ngoroutine 473571 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c36d6e0)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc208dfc600, 0x1, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc208f1c780, 0x5447b68, 0xc208dfc600, 0x5447bc0, 0xc2090ea380, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:599 +0x624\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc208f1c780, 0x5447b68, 0xc208dfc600, 0x5447bc0, 0xc2090ea380, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc208ffb570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc2096ae790, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc208ffb7b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc208f1c780, 0x5447b68, 0xc208dfc600, 0x5447bc0, 0xc2090ea380, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc208f1c720, 0x5447b68, 0xc208dfc600, 0x5447bc0, 0xc2090ea380, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc208dfc600, 0x5447bc0, 0xc2090ea380, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc208dfc600, 0xc2090ea380, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc208ffbf28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc208ffbf28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc208aa2d00, 0x4ae75e0, 0xc208dfc600, 0x16, 0x4add300, 0xc2090ea380, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 470088 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c368d20)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc208f49000, 0x1, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc208262a50, 0x5447b68, 0xc208f49000, 0x5447bc0, 0xc209300f50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:599 +0x624\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc208262a50, 0x5447b68, 0xc208f49000, 0x5447bc0, 0xc209300f50, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc209f67570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc208257930, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc209f677b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc208262a50, 0x5447b68, 0xc208f49000, 0x5447bc0, 0xc209300f50, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc2082629f0, 0x5447b68, 0xc208f49000, 0x5447bc0, 0xc209300f50, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc208f49000, 0x5447bc0, 0xc209300f50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc208f49000, 0xc209300f50, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc209f67f28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc209f67f28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc209e86600, 0x4ae75e0, 0xc208f49000, 0x16, 0x4add300, 0xc209300f50, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n\r\ngoroutine 471809 [semacquire]:\r\nsync.(*WaitGroup).Wait(0xc20c598980)\r\n\t/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).beginCmd(0xc208160090, 0xc208440400, 0xc208042701, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:527 +0x16b\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).addReadOnlyCmd(0xc208160090, 0x5439b80, 0xc20a427aa0, 0x5447b68, 0xc208440400, 0x5447bc0, 0xc2081fe9a0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:599 +0x624\r\ngithub.com/cockroachdb/cockroach/storage.(*Range).AddCmd(0xc208160090, 0x5439b80, 0xc20a427aa0, 0x5447b68, 0xc208440400, 0x5447bc0, 0xc2081fe9a0, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/range.go:510 +0x4a1\r\ngithub.com/cockroachdb/cockroach/storage.func·042(0xc2097c9570, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1249 +0x1e8\r\ngithub.com/cockroachdb/cockroach/util/retry.WithBackoff(0xc20b129db0, 0xa, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x1, 0x0, 0xc2097c97b0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/util/retry/retry.go:86 +0x9e\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc20815e000, 0x5439b80, 0xc20a427aa0, 0x5447b68, 0xc208440400, 0x5447bc0, 0xc2081fe9a0, 0x0, 0x0, 0x0, ...)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/storage/store.go:1287 +0x687\r\ngithub.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc20801d780, 0x5439b80, 0xc20a427a40, 0x5447b68, 0xc208440400, 0x5447bc0, 0xc2081fe9a0, 0x0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:156 +0x514\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).executeCmd(0xc20806e500, 0x5447b68, 0xc208440400, 0x5447bc0, 0xc2081fe9a0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:559 +0xd3\r\ngithub.com/cockroachdb/cockroach/server.(*nodeServer).Get(0xc20806e500, 0xc208440400, 0xc2081fe9a0, 0x0, 0x0)\r\n\t/Users/bram/go/src/github.com/cockroachdb/cockroach/server/node.go:565 +0xb1\r\nreflect.Value.call(0x49d7280, 0x4aebac0, 0x13, 0x4b16390, 0x4, 0xc2097c9f28, 0x3, 0x3, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/reflect/value.go:419 +0x10e5\r\nreflect.Value.Call(0x49d7280, 0x4aebac0, 0x13, 0xc2097c9f28, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/reflect/value.go:296 +0xbc\r\nnet/rpc.(*service).call(0xc2080fc500, 0xc2080fc200, 0xc208232618, 0xc208061800, 0xc20ad1ee20, 0x4ae75e0, 0xc208440400, 0x16, 0x4add300, 0xc2081fe9a0, ...)\r\n\t/usr/local/go/src/net/rpc/server.go:382 +0x1f7\r\ncreated by net/rpc.(*Server).ServeCodec\r\n\t/usr/local/go/src/net/rpc/server.go:476 +0x44a\r\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/115378114","html_url":"https://github.com/cockroachdb/cockroach/issues/1483#issuecomment-115378114","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/1483","id":115378114,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2015-06-25T20:03:15Z","updated_at":"2015-06-25T20:03:15Z","body":"are you 100% sure you didn't have Ben's commit? This response looks like it never went through an error-free run of `InternalPushTxn`, it's exactly what you'd expect with the bug he fixed. Try verbose-printing the reply whenever we hit the response cache."}}