{"action":"created","comment":{"url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/comments/31964973","id":31964973,"diff_hunk":"@@ -0,0 +1,79 @@\n+/*\n+ * Licensed to Elasticsearch under one or more contributor\n+ * license agreements. See the NOTICE file distributed with\n+ * this work for additional information regarding copyright\n+ * ownership. Elasticsearch licenses this file to you under\n+ * the Apache License, Version 2.0 (the \"License\"); you may\n+ * not use this file except in compliance with the License.\n+ * You may obtain a copy of the License at\n+ *\n+ *    http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing,\n+ * software distributed under the License is distributed on an\n+ * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n+ * KIND, either express or implied.  See the License for the\n+ * specific language governing permissions and limitations\n+ * under the License.\n+ */\n+package org.elasticsearch.cluster.routing.allocation.allocator.probabilistic;\n+\n+import org.elasticsearch.cluster.routing.allocation.decider.ConcurrentRebalanceAllocationDecider;\n+\n+public class ModelBalancer {\n+    private ModelCluster currentCluster;\n+    private ModelCluster bestCluster;\n+\n+    /**\n+     * Attempts to find a cluster up to CLUSTER_ROUTING_ALLOCATION_CLUSTER_CONCURRENT_REBALANCE operations away whose\n+     * energy is lower than the current cluster.\n+     * @param cluster\n+     * @return\n+     */\n+    public ModelCluster balance(final ModelCluster cluster) {\n+        int maxIterations = cluster.getSettings().getAsInt(LocalMinimumShardsAllocator.SETTING_MAX_FORKING_ITERATIONS, cluster.getNumNodes() * cluster.getNumShards());\n+        int maxConcurrentRebalanceOperations = cluster.getSettings().getAsInt(ConcurrentRebalanceAllocationDecider.CLUSTER_ROUTING_ALLOCATION_CLUSTER_CONCURRENT_REBALANCE, 2);\n+        maxConcurrentRebalanceOperations = maxConcurrentRebalanceOperations == -1 ? Integer.MAX_VALUE : maxConcurrentRebalanceOperations;\n+\n+        bestCluster = new ModelCluster(cluster);\n+        bestCluster.getForkingOperationHistory().clear();\n+        double bestEnergy = evaluateBalance(bestCluster);\n+        double currentEnergy;\n+        for (int i = 0; i < maxIterations; i++) {\n+            currentCluster = bestCluster.rebalance();\n+            if (currentCluster == null) {","path":"src/main/java/org/elasticsearch/cluster/routing/allocation/allocator/probabilistic/ModelBalancer.java","position":44,"original_position":44,"commit_id":"ca6d286eb320a2672f89ec491e00a26ecd0d55d9","original_commit_id":"ca6d286eb320a2672f89ec491e00a26ecd0d55d9","user":{"login":"sawyercade","id":2049712,"avatar_url":"https://avatars.githubusercontent.com/u/2049712?v=3","gravatar_id":"","url":"https://api.github.com/users/sawyercade","html_url":"https://github.com/sawyercade","followers_url":"https://api.github.com/users/sawyercade/followers","following_url":"https://api.github.com/users/sawyercade/following{/other_user}","gists_url":"https://api.github.com/users/sawyercade/gists{/gist_id}","starred_url":"https://api.github.com/users/sawyercade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sawyercade/subscriptions","organizations_url":"https://api.github.com/users/sawyercade/orgs","repos_url":"https://api.github.com/users/sawyercade/repos","events_url":"https://api.github.com/users/sawyercade/events{/privacy}","received_events_url":"https://api.github.com/users/sawyercade/received_events","type":"User","site_admin":false},"body":"Yeah, it's a placeholder for when no changes can be made to the cluster (super rare). I'll refactor to use ModelCluster.NO_CHANGES or something similar.","created_at":"2015-06-08T22:06:21Z","updated_at":"2015-06-08T22:06:21Z","html_url":"https://github.com/TTAGG/elasticsearch/pull/1#discussion_r31964973","pull_request_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1","_links":{"self":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/comments/31964973"},"html":{"href":"https://github.com/TTAGG/elasticsearch/pull/1#discussion_r31964973"},"pull_request":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1"}}},"pull_request":{"url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1","id":37189354,"html_url":"https://github.com/TTAGG/elasticsearch/pull/1","diff_url":"https://github.com/TTAGG/elasticsearch/pull/1.diff","patch_url":"https://github.com/TTAGG/elasticsearch/pull/1.patch","issue_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/1","number":1,"state":"open","locked":false,"title":"2926 annealing allocator","user":{"login":"sawyercade","id":2049712,"avatar_url":"https://avatars.githubusercontent.com/u/2049712?v=3","gravatar_id":"","url":"https://api.github.com/users/sawyercade","html_url":"https://github.com/sawyercade","followers_url":"https://api.github.com/users/sawyercade/followers","following_url":"https://api.github.com/users/sawyercade/following{/other_user}","gists_url":"https://api.github.com/users/sawyercade/gists{/gist_id}","starred_url":"https://api.github.com/users/sawyercade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sawyercade/subscriptions","organizations_url":"https://api.github.com/users/sawyercade/orgs","repos_url":"https://api.github.com/users/sawyercade/repos","events_url":"https://api.github.com/users/sawyercade/events{/privacy}","received_events_url":"https://api.github.com/users/sawyercade/received_events","type":"User","site_admin":false},"body":"@aewhite @kennycason \r\n\r\nLocalMinimumShardsAllocator attempts to balance a cluster based on the ratio of the maximum node size to the minimum node size. Balancing with this strategy makes it far less likely that we run out of memory due to the field data cache.\r\n\r\nThere are three primary operations handled by the allocator:\r\n1. Allocate unassigned shards - LocalMinimumShardsAllocator always allocates unassigned shards to the smallest valid node.\r\n2. Rebalance the cluster - for this operation, LocalMinimumShardsAllocator builds a model of the cluster, and then repeatedly forks the cluster by randomly selecting a shard to move and the node to move it to. If the move results in a more balanced model cluster (that is, maxNode / minNode is reduced), then the operation is performed on the real ES cluster.\r\n3. Move a shard from a node - in some cases, a shard cannot remain on a node, and must be moved elsewhere. This method attempts to find a new node for the shard to reside on, and moves the shard to the first valid node it finds. Rebalance is called afterwards.\r\n\r\nNote that there's no use of simulated annealing as originally planned, as the current methods performed better on almost all accounts (runtime, number of required move operations, stability, and end balance).\r\n\r\nTo see it in action, run LocalMinimumShardsAllocatorTests. However, note that this test creates a cluster with a random number of nodes and shards, so while it's very likely that the end cluster will have a balance below the specified 1.5x threshold, it's not guaranteed that this test will pass.","created_at":"2015-06-08T19:53:55Z","updated_at":"2015-06-08T22:06:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"58e8a663391d379738dd4aa9f0d9a9e5a01632ff","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1/commits","review_comments_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1/comments","review_comment_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/comments{/number}","comments_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/1/comments","statuses_url":"https://api.github.com/repos/TTAGG/elasticsearch/statuses/ca6d286eb320a2672f89ec491e00a26ecd0d55d9","head":{"label":"TTAGG:2926-annealing-allocator","ref":"2926-annealing-allocator","sha":"ca6d286eb320a2672f89ec491e00a26ecd0d55d9","user":{"login":"TTAGG","id":3141241,"avatar_url":"https://avatars.githubusercontent.com/u/3141241?v=3","gravatar_id":"","url":"https://api.github.com/users/TTAGG","html_url":"https://github.com/TTAGG","followers_url":"https://api.github.com/users/TTAGG/followers","following_url":"https://api.github.com/users/TTAGG/following{/other_user}","gists_url":"https://api.github.com/users/TTAGG/gists{/gist_id}","starred_url":"https://api.github.com/users/TTAGG/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTAGG/subscriptions","organizations_url":"https://api.github.com/users/TTAGG/orgs","repos_url":"https://api.github.com/users/TTAGG/repos","events_url":"https://api.github.com/users/TTAGG/events{/privacy}","received_events_url":"https://api.github.com/users/TTAGG/received_events","type":"Organization","site_admin":false},"repo":{"id":35570818,"name":"elasticsearch","full_name":"TTAGG/elasticsearch","owner":{"login":"TTAGG","id":3141241,"avatar_url":"https://avatars.githubusercontent.com/u/3141241?v=3","gravatar_id":"","url":"https://api.github.com/users/TTAGG","html_url":"https://github.com/TTAGG","followers_url":"https://api.github.com/users/TTAGG/followers","following_url":"https://api.github.com/users/TTAGG/following{/other_user}","gists_url":"https://api.github.com/users/TTAGG/gists{/gist_id}","starred_url":"https://api.github.com/users/TTAGG/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTAGG/subscriptions","organizations_url":"https://api.github.com/users/TTAGG/orgs","repos_url":"https://api.github.com/users/TTAGG/repos","events_url":"https://api.github.com/users/TTAGG/events{/privacy}","received_events_url":"https://api.github.com/users/TTAGG/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/TTAGG/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":true,"url":"https://api.github.com/repos/TTAGG/elasticsearch","forks_url":"https://api.github.com/repos/TTAGG/elasticsearch/forks","keys_url":"https://api.github.com/repos/TTAGG/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/TTAGG/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/TTAGG/elasticsearch/teams","hooks_url":"https://api.github.com/repos/TTAGG/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/TTAGG/elasticsearch/events","assignees_url":"https://api.github.com/repos/TTAGG/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/TTAGG/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/TTAGG/elasticsearch/tags","blobs_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/TTAGG/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/TTAGG/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/TTAGG/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/TTAGG/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/TTAGG/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/TTAGG/elasticsearch/subscription","commits_url":"https://api.github.com/repos/TTAGG/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/TTAGG/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/TTAGG/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/TTAGG/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/TTAGG/elasticsearch/merges","archive_url":"https://api.github.com/repos/TTAGG/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/TTAGG/elasticsearch/downloads","issues_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/TTAGG/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/TTAGG/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/TTAGG/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/TTAGG/elasticsearch/releases{/id}","created_at":"2015-05-13T19:48:08Z","updated_at":"2015-05-13T19:48:27Z","pushed_at":"2015-06-08T19:53:55Z","git_url":"git://github.com/TTAGG/elasticsearch.git","ssh_url":"git@github.com:TTAGG/elasticsearch.git","clone_url":"https://github.com/TTAGG/elasticsearch.git","svn_url":"https://github.com/TTAGG/elasticsearch","homepage":"http://elasticsearch.org","size":149776,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"TTAGG:master","ref":"master","sha":"33fd25083e520e40cc9181a0d690e637b6eb04f4","user":{"login":"TTAGG","id":3141241,"avatar_url":"https://avatars.githubusercontent.com/u/3141241?v=3","gravatar_id":"","url":"https://api.github.com/users/TTAGG","html_url":"https://github.com/TTAGG","followers_url":"https://api.github.com/users/TTAGG/followers","following_url":"https://api.github.com/users/TTAGG/following{/other_user}","gists_url":"https://api.github.com/users/TTAGG/gists{/gist_id}","starred_url":"https://api.github.com/users/TTAGG/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTAGG/subscriptions","organizations_url":"https://api.github.com/users/TTAGG/orgs","repos_url":"https://api.github.com/users/TTAGG/repos","events_url":"https://api.github.com/users/TTAGG/events{/privacy}","received_events_url":"https://api.github.com/users/TTAGG/received_events","type":"Organization","site_admin":false},"repo":{"id":35570818,"name":"elasticsearch","full_name":"TTAGG/elasticsearch","owner":{"login":"TTAGG","id":3141241,"avatar_url":"https://avatars.githubusercontent.com/u/3141241?v=3","gravatar_id":"","url":"https://api.github.com/users/TTAGG","html_url":"https://github.com/TTAGG","followers_url":"https://api.github.com/users/TTAGG/followers","following_url":"https://api.github.com/users/TTAGG/following{/other_user}","gists_url":"https://api.github.com/users/TTAGG/gists{/gist_id}","starred_url":"https://api.github.com/users/TTAGG/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTAGG/subscriptions","organizations_url":"https://api.github.com/users/TTAGG/orgs","repos_url":"https://api.github.com/users/TTAGG/repos","events_url":"https://api.github.com/users/TTAGG/events{/privacy}","received_events_url":"https://api.github.com/users/TTAGG/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/TTAGG/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":true,"url":"https://api.github.com/repos/TTAGG/elasticsearch","forks_url":"https://api.github.com/repos/TTAGG/elasticsearch/forks","keys_url":"https://api.github.com/repos/TTAGG/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/TTAGG/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/TTAGG/elasticsearch/teams","hooks_url":"https://api.github.com/repos/TTAGG/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/TTAGG/elasticsearch/events","assignees_url":"https://api.github.com/repos/TTAGG/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/TTAGG/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/TTAGG/elasticsearch/tags","blobs_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/TTAGG/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/TTAGG/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/TTAGG/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/TTAGG/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/TTAGG/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/TTAGG/elasticsearch/subscription","commits_url":"https://api.github.com/repos/TTAGG/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/TTAGG/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/TTAGG/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/TTAGG/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/TTAGG/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/TTAGG/elasticsearch/merges","archive_url":"https://api.github.com/repos/TTAGG/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/TTAGG/elasticsearch/downloads","issues_url":"https://api.github.com/repos/TTAGG/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/TTAGG/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/TTAGG/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/TTAGG/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/TTAGG/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/TTAGG/elasticsearch/releases{/id}","created_at":"2015-05-13T19:48:08Z","updated_at":"2015-05-13T19:48:27Z","pushed_at":"2015-06-08T19:53:55Z","git_url":"git://github.com/TTAGG/elasticsearch.git","ssh_url":"git@github.com:TTAGG/elasticsearch.git","clone_url":"https://github.com/TTAGG/elasticsearch.git","svn_url":"https://github.com/TTAGG/elasticsearch","homepage":"http://elasticsearch.org","size":149776,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1"},"html":{"href":"https://github.com/TTAGG/elasticsearch/pull/1"},"issue":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/issues/1"},"comments":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/TTAGG/elasticsearch/statuses/ca6d286eb320a2672f89ec491e00a26ecd0d55d9"}}}}