{"action":"opened","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/13771","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13771/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13771/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13771/events","html_url":"https://github.com/elastic/elasticsearch/issues/13771","id":108161770,"number":13771,"title":"Shard rebalancing too aggressive when adding new nodes","user":{"login":"avleen","id":539525,"avatar_url":"https://avatars.githubusercontent.com/u/539525?v=3","gravatar_id":"","url":"https://api.github.com/users/avleen","html_url":"https://github.com/avleen","followers_url":"https://api.github.com/users/avleen/followers","following_url":"https://api.github.com/users/avleen/following{/other_user}","gists_url":"https://api.github.com/users/avleen/gists{/gist_id}","starred_url":"https://api.github.com/users/avleen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avleen/subscriptions","organizations_url":"https://api.github.com/users/avleen/orgs","repos_url":"https://api.github.com/users/avleen/repos","events_url":"https://api.github.com/users/avleen/events{/privacy}","received_events_url":"https://api.github.com/users/avleen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-24T16:04:27Z","updated_at":"2015-09-24T16:04:27Z","closed_at":null,"body":"Cluster:\r\n * 3 dedicated masters\r\n * 2 client nodes\r\n * 65 data nodes, each running 2 instances of ES\r\n * ES version 1.5.2, but tests on a smaller 1.7 cluster seem to behave the same way\r\n * Shard balancing based on disk usage\r\n * 64 indices, 1757 primary shards, 1757 replica shards (3514 shards total)\r\n\r\nUpon adding a new node to the cluster, the expectation is that this node will get populated quickly.\r\nIn reality, we end up with shards rebalancing across the entire cluster.\r\n\r\nExample:\r\nI added `logdb41` to the cluster today, and the result was primary shards getting relocated and replicas being rebuilt everywhere, and very few going to the new node at any one time:\r\n```\r\nindex               shard time   type       stage source_host          target_host\r\nlogstash-2015.08.29 26    268206 relocation index logdb74              logdb52\r\nlogstash-2015.08.23 36    268138 replica    index logdb70              logdb74\r\nlogstash-2015.08.23 15    268340 replica    index logdb26              logdb38\r\nlogstash-2015.08.25 15    267976 replica    index logdb39              logdb26\r\nlogstash-2015.08.25 28    268094 replica    index logdb32              logdb27\r\nlogstash-2015.09.08 6     245140 replica    index logdb48              logdb41\r\nlogstash-2015.08.25 12    268262 relocation index logdb40              logdb65\r\nlogstash-2015.09.19 25    264782 relocation index logdb65              logdb41\r\n```\r\n\r\nDoes the rebalancing algorithm try to get disk usage as even as possible across the entire cluster?\r\nOr does it allow some amount of variance between nodes?\r\nIt feels like the cluster is coming up with a new map of where all the shards should end up, but instead of initially putting as many shards as possible on the new node, it's prioiritising the rebalancing some other way.\r\nThis seems less efficient in terms of production operations. Do all these shards really need to move?\r\n\r\nThanks!"}}