{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/39531977","id":39531977,"diff_hunk":"@@ -273,15 +279,27 @@ func (a allocator) removeTarget(existing []proto.Replica) (proto.Replica, error)\n \treturn worst.repl, nil\n }\n \n-// shouldRebalance returns whether the specified store is overweight\n-// according to the cluster mean and should rebalance a range.\n-func (a allocator) shouldRebalance(s *proto.StoreDescriptor) bool {\n-\tsl := a.storePool.getStoreList(*s.CombinedAttrs(), a.deterministic)\n-\n+// shouldRebalance returns whether the specified store should attempt to\n+// rebalance a replica to another store.\n+func (a allocator) shouldRebalance(storeID proto.StoreID) bool {\n+\t// In production, add some random jitter to shouldRebalance.\n+\tif !a.options.Deterministic && a.randGen.Float32() > rebalanceShouldRebalanceChance {\n+\t\treturn false\n+\t}\n+\tstoreDesc := a.storePool.getStoreDescriptor(storeID)\n+\tif storeDesc == nil {\n+\t\tif log.V(1) {\n+\t\t\tlog.Warningf(\n+\t\t\t\t\"shouldRebalance couldn't find store with id %d in StorePool\",\n+\t\t\t\tstoreID)\n+\t\t}\n+\t\treturn false\n+\t}\n+\tsl := a.storePool.getStoreList(*storeDesc.CombinedAttrs(), a.options.Deterministic)\n \tif sl.used.mean < minFractionUsedThreshold {\n-\t\treturn s.Capacity.RangeCount > int32(math.Ceil(sl.count.mean))\n+\t\treturn storeDesc.Capacity.RangeCount > int32(math.Ceil(sl.count.mean))","path":"storage/allocator.go","position":69,"original_position":69,"commit_id":"91d332bb81a49bc1e40ca3f9463233e92d7a3aa4","original_commit_id":"91d332bb81a49bc1e40ca3f9463233e92d7a3aa4","user":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"body":"Yeah, I think we should be using `rebalanceFromMean` for all of these comparisons to the mean. Right now we won't consider a node to be a rebalance destination unless it is below the mean by `rebalanceFromMean`; we shouldn't consider a node as a rebalance source unless it is also above the mean by `rebalanceFromMean`. ","created_at":"2015-09-15T16:26:44Z","updated_at":"2015-09-15T16:26:44Z","html_url":"https://github.com/cockroachdb/cockroach/pull/2503#discussion_r39531977","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/39531977"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/2503#discussion_r39531977"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503","id":44988828,"html_url":"https://github.com/cockroachdb/cockroach/pull/2503","diff_url":"https://github.com/cockroachdb/cockroach/pull/2503.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/2503.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/2503","number":2503,"state":"open","locked":false,"title":"Add Rebalancing to Replicate Queue","user":{"login":"mrtracy","id":6969858,"avatar_url":"https://avatars.githubusercontent.com/u/6969858?v=3","gravatar_id":"","url":"https://api.github.com/users/mrtracy","html_url":"https://github.com/mrtracy","followers_url":"https://api.github.com/users/mrtracy/followers","following_url":"https://api.github.com/users/mrtracy/following{/other_user}","gists_url":"https://api.github.com/users/mrtracy/gists{/gist_id}","starred_url":"https://api.github.com/users/mrtracy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrtracy/subscriptions","organizations_url":"https://api.github.com/users/mrtracy/orgs","repos_url":"https://api.github.com/users/mrtracy/repos","events_url":"https://api.github.com/users/mrtracy/events{/privacy}","received_events_url":"https://api.github.com/users/mrtracy/received_events","type":"User","site_admin":false},"body":"ReplicateQueue will now attempt to rebalance replicas to other stores in some\r\ncases.\r\n\r\n+ If the ReplicateQueue's `shouldQueue()` detects that no action is *necessary* on a particular\r\nreplica, it calls `allocator.shouldRebalance()` to determine if a rebalancing\r\nopportunity might be wise.\r\n+ If the ReplicateQueue's `process()` method encounters a replica for which no\r\naction is *necessary*, it calls `allocator.rebalanceTarget()`. If that method\r\nreturns a store, a replica is added on that store.\r\n\r\nRebalancing must be enabled via the `RebalancingOptions` on StoreContext;\r\ncurrently, this is set to false on the default server, and is only enabled for\r\nthe rebalancing test.\r\n\r\nFurthermore, typical Rebalance operations include some randomness in decisions\r\nin the anticipation of distributing load better when multiple stores are trying\r\nto rebalance. This randomness can be disabled via the `Deterministic` option,\r\nalso on the RebalancingOptions structure.","created_at":"2015-09-14T22:25:52Z","updated_at":"2015-09-15T16:26:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d8fba85b7f02db2d0e12c9fb408644e025a1c542","assignee":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/2503/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/91d332bb81a49bc1e40ca3f9463233e92d7a3aa4","head":{"label":"mrtracy:mtracy/rebalance_to_replicate_queue","ref":"mtracy/rebalance_to_replicate_queue","sha":"91d332bb81a49bc1e40ca3f9463233e92d7a3aa4","user":{"login":"mrtracy","id":6969858,"avatar_url":"https://avatars.githubusercontent.com/u/6969858?v=3","gravatar_id":"","url":"https://api.github.com/users/mrtracy","html_url":"https://github.com/mrtracy","followers_url":"https://api.github.com/users/mrtracy/followers","following_url":"https://api.github.com/users/mrtracy/following{/other_user}","gists_url":"https://api.github.com/users/mrtracy/gists{/gist_id}","starred_url":"https://api.github.com/users/mrtracy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrtracy/subscriptions","organizations_url":"https://api.github.com/users/mrtracy/orgs","repos_url":"https://api.github.com/users/mrtracy/repos","events_url":"https://api.github.com/users/mrtracy/events{/privacy}","received_events_url":"https://api.github.com/users/mrtracy/received_events","type":"User","site_admin":false},"repo":{"id":39856388,"name":"cockroach","full_name":"mrtracy/cockroach","owner":{"login":"mrtracy","id":6969858,"avatar_url":"https://avatars.githubusercontent.com/u/6969858?v=3","gravatar_id":"","url":"https://api.github.com/users/mrtracy","html_url":"https://github.com/mrtracy","followers_url":"https://api.github.com/users/mrtracy/followers","following_url":"https://api.github.com/users/mrtracy/following{/other_user}","gists_url":"https://api.github.com/users/mrtracy/gists{/gist_id}","starred_url":"https://api.github.com/users/mrtracy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrtracy/subscriptions","organizations_url":"https://api.github.com/users/mrtracy/orgs","repos_url":"https://api.github.com/users/mrtracy/repos","events_url":"https://api.github.com/users/mrtracy/events{/privacy}","received_events_url":"https://api.github.com/users/mrtracy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mrtracy/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":true,"url":"https://api.github.com/repos/mrtracy/cockroach","forks_url":"https://api.github.com/repos/mrtracy/cockroach/forks","keys_url":"https://api.github.com/repos/mrtracy/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mrtracy/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mrtracy/cockroach/teams","hooks_url":"https://api.github.com/repos/mrtracy/cockroach/hooks","issue_events_url":"https://api.github.com/repos/mrtracy/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/mrtracy/cockroach/events","assignees_url":"https://api.github.com/repos/mrtracy/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/mrtracy/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/mrtracy/cockroach/tags","blobs_url":"https://api.github.com/repos/mrtracy/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mrtracy/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mrtracy/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/mrtracy/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mrtracy/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/mrtracy/cockroach/languages","stargazers_url":"https://api.github.com/repos/mrtracy/cockroach/stargazers","contributors_url":"https://api.github.com/repos/mrtracy/cockroach/contributors","subscribers_url":"https://api.github.com/repos/mrtracy/cockroach/subscribers","subscription_url":"https://api.github.com/repos/mrtracy/cockroach/subscription","commits_url":"https://api.github.com/repos/mrtracy/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/mrtracy/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/mrtracy/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/mrtracy/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/mrtracy/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/mrtracy/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mrtracy/cockroach/merges","archive_url":"https://api.github.com/repos/mrtracy/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mrtracy/cockroach/downloads","issues_url":"https://api.github.com/repos/mrtracy/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/mrtracy/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/mrtracy/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/mrtracy/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mrtracy/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/mrtracy/cockroach/releases{/id}","created_at":"2015-07-28T20:33:41Z","updated_at":"2015-07-28T20:33:42Z","pushed_at":"2015-09-14T22:23:08Z","git_url":"git://github.com/mrtracy/cockroach.git","ssh_url":"git@github.com:mrtracy/cockroach.git","clone_url":"https://github.com/mrtracy/cockroach.git","svn_url":"https://github.com/mrtracy/cockroach","homepage":null,"size":28715,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"9bdc4cd779229a4564f250a38915ab98e1f69e3a","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","created_at":"2014-02-06T00:18:47Z","updated_at":"2015-09-15T08:47:56Z","pushed_at":"2015-09-15T16:22:01Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":null,"size":54319,"stargazers_count":5254,"watchers_count":5254,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":426,"mirror_url":null,"open_issues_count":181,"forks":426,"open_issues":181,"watchers":5254,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/2503"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/2503"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/2503/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2503/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/91d332bb81a49bc1e40ca3f9463233e92d7a3aa4"}}}}