{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3038","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3038/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3038/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3038/events","html_url":"https://github.com/cockroachdb/cockroach/issues/3038","id":115623858,"number":3038,"title":"flakiness in TestSplitSnapshotRace_SnapshotWins","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-11-07T01:48:20Z","updated_at":"2015-11-11T22:09:26Z","closed_at":null,"body":"An unhandled `ConditionFailedError` on a replica change.\r\n\r\n```\r\n...\r\nE1106 20:42:23.832435 28896 storage/client_test.go:541  engine 0: missing key \"z\"\r\nE1106 20:42:23.832494 28896 storage/client_test.go:541  engine 4: missing key \"z\"\r\nE1106 20:42:23.832518 28896 storage/client_test.go:541  engine 5: missing key \"z\"\r\nW1106 20:42:23.833490 28896 multiraft/multiraft.go:1226  node 3 failed to send message to 4: multiraft/transport.go:143: unknown peer 4\r\nW1106 20:42:23.833984 28896 multiraft/multiraft.go:1226  node 3 failed to send message to 4: multiraft/transport.go:143: unknown peer 4\r\nI1106 20:42:23.834840 28896 storage/replica_command.go:1191  initiating a split of range=2 [\"\\x01\"-\"\\xff\\xff\") at key \"m\"\r\n--- FAIL: TestSplitSnapshotRace_SnapshotWins (0.48s)\r\n\tclient_test.go:490: storage/replica_command.go:1588: change replicas of 2 failed: unexpected value: raw_bytes:\"\\010\\002\\022\\001\\001\\032\\001m\\\"\\006\\010\\004\\020\\004\\030\\004\\\"\\006\\010\\002\\020\\002\\030\\002\\\"\\006\\010\\003\\020\\003\\030\\003(\\005\" checksum:3738718210 timestamp:<wall_time:2000000002 logical:12 > tag:BYTES\r\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/155925424","html_url":"https://github.com/cockroachdb/cockroach/issues/3038#issuecomment-155925424","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3038","id":155925424,"user":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"created_at":"2015-11-11T22:09:26Z","updated_at":"2015-11-11T22:09:26Z","body":"After talking it over with @BramGruneir, here's the plan:\r\n* Remove `raftGroupLocker`. Keep the `GroupLocker` method (probably under a new name), but have it return `store.mu` instead.\r\n* `multiraft` acquires `store.mu` before calling any methods on the `Storage` interface.\r\n* `Store` methods that implement the `multiraft.Storage` interface can assume `store.mu` is held and no longer acquire this lock for themselves. New pre-locked versions of various methods will be added as necessary."}}