{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/12926","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12926/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12926/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12926/events","html_url":"https://github.com/elastic/elasticsearch/issues/12926","id":101375198,"number":12926,"title":"Shards stuck in INITIALIZING","user":{"login":"krisb78","id":222033,"avatar_url":"https://avatars.githubusercontent.com/u/222033?v=3","gravatar_id":"","url":"https://api.github.com/users/krisb78","html_url":"https://github.com/krisb78","followers_url":"https://api.github.com/users/krisb78/followers","following_url":"https://api.github.com/users/krisb78/following{/other_user}","gists_url":"https://api.github.com/users/krisb78/gists{/gist_id}","starred_url":"https://api.github.com/users/krisb78/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krisb78/subscriptions","organizations_url":"https://api.github.com/users/krisb78/orgs","repos_url":"https://api.github.com/users/krisb78/repos","events_url":"https://api.github.com/users/krisb78/events{/privacy}","received_events_url":"https://api.github.com/users/krisb78/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-08-17T09:11:46Z","updated_at":"2015-08-17T13:02:04Z","closed_at":"2015-08-17T12:54:37Z","body":"I have a fairly large index (~3.5 million documents, with nested fields, some of the docs are quite large) where the documents are being updated a lot.\r\n\r\nA while ago the cluster went into the yellow state and I'm seeing a number of shards stuck in the \"INITIALIZING\" state:\r\n\r\n```\r\njoint_user_summary_v1                   0 r INITIALIZING                   127.0.1.1 cubitsearch-1\r\njoint_user_summary_v1                   7 r INITIALIZING                   127.0.1.1 cubitsearch-5\r\njoint_user_summary_v1                   7 r INITIALIZING                   127.0.1.1 cubitsearch-2\r\njoint_user_summary_v1                   3 r INITIALIZING                   127.0.1.1 cubitsearch-1\r\njoint_user_summary_v1                   1 r INITIALIZING                   127.0.1.1 cubitsearch-4\r\njoint_user_summary_v1                   1 r INITIALIZING                   127.0.1.1 cubitsearch-3\r\njoint_user_summary_v1                   5 r INITIALIZING                   127.0.1.1 cubitsearch-2\r\njoint_user_summary_v1                   6 r INITIALIZING                   127.0.1.1 cubitsearch-3\r\njoint_user_summary_v1                   6 r INITIALIZING                   127.0.1.1 cubitsearch-5\r\n```\r\n\r\nNaturally, I can't optimize this index anymore:\r\n\r\n```\r\n{\"_shards\":{\"total\":30,\"successful\":5,\"failed\":6,\"failures\":[{\"index\":\"joint_user_summary_v1\",\"shard\":0,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][0] ]; nested: RemoteTransportException[[cubitsearch-2][inet[/10.0.1.236:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][0] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][0] recovery is in progress, flush is not allowed]; \"},{\"index\":\"joint_user_summary_v1\",\"shard\":1,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][1] ]; nested: RemoteTransportException[[cubitsearch-2][inet[/10.0.1.236:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][1] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][1] recovery is in progress, flush is not allowed]; \"},{\"index\":\"joint_user_summary_v1\",\"shard\":3,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][3] ]; nested: RemoteTransportException[[cubitsearch-3][inet[/10.0.1.237:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][3] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][3] recovery is in progress, flush is not allowed]; \"},{\"index\":\"joint_user_summary_v1\",\"shard\":5,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][5] ]; nested: RemoteTransportException[[cubitsearch-1][inet[/10.0.1.235:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][5] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][5] recovery is in progress, flush is not allowed]; \"},{\"index\":\"joint_user_summary_v1\",\"shard\":6,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][6] ]; nested: RemoteTransportException[[cubitsearch-1][inet[/10.0.1.235:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][6] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][6] recovery is in progress, flush is not allowed]; \"},{\"index\":\"joint_user_summary_v1\",\"shard\":7,\"status\":500,\"reason\":\"BroadcastShardOperationFailedException[[joint_user_summary_v1][7] ]; nested: RemoteTransportException[[cubitsearch-3][inet[/10.0.1.237:9300]][indices:admin/optimize[s]]]; nested: OptimizeFailedEngineException[[joint_user_summary_v1][7] force merge failed]; nested: FlushNotAllowedEngineException[[joint_user_summary_v1][7] recovery is in progress, flush is not allowed]; \"}]}}\r\n```\r\n\r\n\r\nHere's an excerpt from the log file that may shed some light on the issue:\r\n\r\n```\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: [2015-08-17 09:06:31,525][DEBUG][action.bulk              ] [cubitsearch-3] observer: timeout notification from cluster service. timeout setting [60ms], time since start [60ms]\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: [2015-08-17 09:06:31,525][DEBUG][action.bulk              ] [cubitsearch-3] observer: timeout notification from cluster service. timeout setting [60ms], time since start [60ms]\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: [2015-08-17 09:06:31,939][WARN ][indices.cluster          ] [cubitsearch-3] [[joint_user_summary_v1][1]] marking and sending shard failed due to [failed recovery]\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: org.elasticsearch.indices.recovery.RecoveryFailedException: [joint_user_summary_v1][1]: Recovery failed from [cubitsearch-2][U2eMGyQ7Rhic1lHqzIoLXA][cubitsearch-2][inet[/10.0.1.236:9300]]{max_local_storage_nodes=1, master=false} into [cubitsearch-3][IEP7xQczReKinA78HfLC3Q][cubitsearch-3][inet[/10.0.1.237:9300]]{max_local_storage_nodes=1, master=false}\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:561)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.lang.Thread.run(Thread.java:745)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: Caused by: org.elasticsearch.transport.RemoteTransportException: [cubitsearch-2][inet[/10.0.1.236:9300]][internal:index/shard/recovery/start_recovery]\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [joint_user_summary_v1][1] Phase[2] Execution failed\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:902)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:780)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at java.lang.Thread.run(Thread.java:745)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [cubitsearch-3][inet[/10.0.1.237:9300]][internal:index/shard/recovery/prepare_translog] request_id [14280416] timed out after [900000ms]\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)\r\nAug 17 09:06:31 cubitsearch-3 elasticsearch: #011... 3 more\r\n```\r\n\r\nI was hoping that upgrading to the latest version (1.7.1) would fix the problem, but unfortunately it persists.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/131810371","html_url":"https://github.com/elastic/elasticsearch/issues/12926#issuecomment-131810371","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12926","id":131810371,"user":{"login":"krisb78","id":222033,"avatar_url":"https://avatars.githubusercontent.com/u/222033?v=3","gravatar_id":"","url":"https://api.github.com/users/krisb78","html_url":"https://github.com/krisb78","followers_url":"https://api.github.com/users/krisb78/followers","following_url":"https://api.github.com/users/krisb78/following{/other_user}","gists_url":"https://api.github.com/users/krisb78/gists{/gist_id}","starred_url":"https://api.github.com/users/krisb78/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krisb78/subscriptions","organizations_url":"https://api.github.com/users/krisb78/orgs","repos_url":"https://api.github.com/users/krisb78/repos","events_url":"https://api.github.com/users/krisb78/events{/privacy}","received_events_url":"https://api.github.com/users/krisb78/received_events","type":"User","site_admin":false},"created_at":"2015-08-17T13:02:04Z","updated_at":"2015-08-17T13:02:04Z","body":"I did restart all of them a couple of times, but it didn't help...\r\n\r\nThanks for your prompt responses though, I'll be keeping an eye on https://github.com/elastic/elasticsearch/issues/12011"}}