{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14927","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14927/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14927/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14927/events","html_url":"https://github.com/elastic/elasticsearch/issues/14927","id":118339148,"number":14927,"title":"After upgrading from ES 1.7 to 2.0, getting lots of IllegalIndexShardStateException when bulk creating indices","user":{"login":"bjorn-ali-goransson","id":668160,"avatar_url":"https://avatars.githubusercontent.com/u/668160?v=3","gravatar_id":"","url":"https://api.github.com/users/bjorn-ali-goransson","html_url":"https://github.com/bjorn-ali-goransson","followers_url":"https://api.github.com/users/bjorn-ali-goransson/followers","following_url":"https://api.github.com/users/bjorn-ali-goransson/following{/other_user}","gists_url":"https://api.github.com/users/bjorn-ali-goransson/gists{/gist_id}","starred_url":"https://api.github.com/users/bjorn-ali-goransson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjorn-ali-goransson/subscriptions","organizations_url":"https://api.github.com/users/bjorn-ali-goransson/orgs","repos_url":"https://api.github.com/users/bjorn-ali-goransson/repos","events_url":"https://api.github.com/users/bjorn-ali-goransson/events{/privacy}","received_events_url":"https://api.github.com/users/bjorn-ali-goransson/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"jasontedor","id":4744941,"avatar_url":"https://avatars.githubusercontent.com/u/4744941?v=3","gravatar_id":"","url":"https://api.github.com/users/jasontedor","html_url":"https://github.com/jasontedor","followers_url":"https://api.github.com/users/jasontedor/followers","following_url":"https://api.github.com/users/jasontedor/following{/other_user}","gists_url":"https://api.github.com/users/jasontedor/gists{/gist_id}","starred_url":"https://api.github.com/users/jasontedor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasontedor/subscriptions","organizations_url":"https://api.github.com/users/jasontedor/orgs","repos_url":"https://api.github.com/users/jasontedor/repos","events_url":"https://api.github.com/users/jasontedor/events{/privacy}","received_events_url":"https://api.github.com/users/jasontedor/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-11-23T08:47:12Z","updated_at":"2015-11-23T19:51:57Z","closed_at":"2015-11-23T18:03:52Z","body":"Hello,\r\n\r\nWe have a tool that creates a language-contenttype matrix of indices at first run (it checks what indices are missing from the matrix), which functioned well using ES 1.7 but now we're getting lots of  `IllegalIndexShardStateException` exceptions.\r\n\r\nSo we issue a `_cluster/health?wait_for_status=yellow` before each index creation HTTP request, still same issue.\r\n\r\nWe wait 1000ms before the cluster health request, and 1000ms after each index creation request, still same issue.\r\n\r\nCould it be that the yellow status is reached too fast? We always run with an empty ES instance when testing (ie remove `data` and `logs` dir before starting). Sometimes even the first index to be created seems to cause the error?\r\n\r\nTips for further investigation is appreciated.\r\n\r\n    [2015-11-23 07:37:58,716][INFO ][cluster.metadata         ] [my-name] [content__1fzc2hfheucqdv2_q5nmpa__en] creating index, cause [api], templates [], shards [5]/[1], mappings [content]\r\n    [2015-11-23 07:38:00,983][INFO ][cluster.metadata         ] [my-name] [content__p9p2lcwchuwe4atddvvd2w__en] creating index, cause [api], templates [], shards [5]/[1], mappings [content]\r\n    [2015-11-23 07:38:03,281][INFO ][cluster.metadata         ] [my-name] [content__tuslngocl06jrzeywjpdzw__en] creating index, cause [api], templates [], shards [5]/[1], mappings [content]\r\n    [2015-11-23 07:38:05,576][INFO ][cluster.metadata         ] [my-name] [content__hkwufjcos0urbpkg6t7dnq__en] creating index, cause [api], templates [], shards [5]/[1], mappings [content]\r\n    [2015-11-23 07:38:07,968][INFO ][cluster.metadata         ] [my-name] [content__w6mzamir00cz2uk6ot7ijw__en] creating index, cause [api], templates [], shards [5]/[1], mappings [content]\r\n    [2015-11-23 07:38:08,299][DEBUG][action.admin.indices.stats] [my-name] [indices:monitor/stats] failed to execute operation for shard [[content__w6mzamir00cz2uk6ot7ijw__en][3], node[NSXOyiZxTT63LKBFnm3ESA], [P], v[1], s[INITIALIZING], a[id=OckfAhmIRiSxOjXG8J4Mcg], unassigned_info[[reason=INDEX_CREATED], at[2015-11-23T06:38:07.969Z]]]\r\n    [content__w6mzamir00cz2uk6ot7ijw__en][[content__w6mzamir00cz2uk6ot7ijw__en][3]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];\r\n      at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:399)\r\n      at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:376)\r\n      at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:365)\r\n      at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)\r\n      at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n      at java.lang.Thread.run(Thread.java:745)\r\n    Caused by: [content__w6mzamir00cz2uk6ot7ijw__en][[content__w6mzamir00cz2uk6ot7ijw__en][3]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]\r\n      at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:957)\r\n      at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:791)\r\n      at org.elasticsearch.index.shard.IndexShard.docStats(IndexShard.java:612)\r\n      at org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:131)\r\n      at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:165)\r\n      at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)\r\n      at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:395)\r\n      ... 7 more\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/159044280","html_url":"https://github.com/elastic/elasticsearch/issues/14927#issuecomment-159044280","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14927","id":159044280,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T19:51:57Z","updated_at":"2015-11-23T19:51:57Z","body":"Oh , I see. OK :) before we didn't even try... \r\n\r\nAs I said, I think we can be less strict then readAllowed() but that's not a regression then. Re the trace - note how we ignore this error completely in the response:\r\n\r\n```\r\n if (!TransportActions.isShardNotAvailableException(throwable)) {\r\n                        exceptions.add(new DefaultShardOperationFailedException(throwable.getIndex(), throwable.getShardId().getId(), throwable));\r\n                    }\r\n```"}}