{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4573","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4573/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4573/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4573/events","html_url":"https://github.com/influxdb/influxdb/issues/4573","id":113345167,"number":4573,"title":"One of the nodes in the cluster not coming up after restarting","user":{"login":"priyadarsh","id":5320320,"avatar_url":"https://avatars.githubusercontent.com/u/5320320?v=3","gravatar_id":"","url":"https://api.github.com/users/priyadarsh","html_url":"https://github.com/priyadarsh","followers_url":"https://api.github.com/users/priyadarsh/followers","following_url":"https://api.github.com/users/priyadarsh/following{/other_user}","gists_url":"https://api.github.com/users/priyadarsh/gists{/gist_id}","starred_url":"https://api.github.com/users/priyadarsh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/priyadarsh/subscriptions","organizations_url":"https://api.github.com/users/priyadarsh/orgs","repos_url":"https://api.github.com/users/priyadarsh/repos","events_url":"https://api.github.com/users/priyadarsh/events{/privacy}","received_events_url":"https://api.github.com/users/priyadarsh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-26T12:31:17Z","updated_at":"2015-10-26T19:48:22Z","closed_at":null,"body":"I have restarted one of the 3 nodes in the cluster set up and after restarting one of the nodes I see the below log for a while and the server dies.\r\n\r\nBuild : 0.9.5-nightly-827c513\r\n\r\n\r\n[monitor] 2015/10/26 17:53:57 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:07 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:17 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:27 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:37 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:47 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:54:57 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:07 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:17 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:27 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:37 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:47 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:55:57 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:56:07 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:56:17 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:56:27 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:56:37 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:56:47 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:57:03 failed to store statistics: timeout\r\n[monitor] 2015/10/26 17:57:08 failed to store statistics: timeout\r\n[tsm1wal] 2015/10/26 17:57:14 failed to flush to index: snappy: corrupt input\r\n[registration] 2015/10/26 17:57:14 registration service terminating\r\n[retention] 2015/10/26 17:57:14 retention policy enforcement terminating\r\n[monitor] 2015/10/26 17:57:14 shutting down monitor system\r\n[monitor] 2015/10/26 17:57:14 terminating storage of statistics\r\n[subscriber] 2015/10/26 17:57:14 closed service\r\nrun: open server: open tsdb store: failed to open shard 5: open engine: snappy: corrupt input\r\n\r\n\r\nAnd in the other 2 nodes, I see the below logs(masked IPs). The  I am trying to restart is on XXX.XX.11.54\r\n\r\n\r\nhandoff] 2015/10/26 18:03:57 remote write failed: read message type: read tcp XXX.XX.11.55:58491->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:03:59 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:03 remote write failed: read message type: read tcp XXX.XX.11.55:58496->XXX.XX.11.54:8088: i/o timeout\r\n[handoff] 2015/10/26 18:04:09 remote write failed: read message type: read tcp XXX.XX.11.55:58498->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:04:09 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:15 remote write failed: read message type: read tcp XXX.XX.11.55:58501->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:04:19 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:21 remote write failed: read message type: read tcp XXX.XX.11.55:58504->XXX.XX.11.54:8088: i/o timeout\r\n[handoff] 2015/10/26 18:04:27 remote write failed: read message type: read tcp XXX.XX.11.55:58506->XXX.XX.11.54:8088: i/o timeout\r\n[\r\n\r\n\r\n\r\n\r\nhandoff] 2015/10/26 18:04:15 remote write failed: read message type: read tcp XXX.XX.11.56:52218->XXX.XX.11.54:8088: i/o timeout\r\n[handoff] 2015/10/26 18:04:21 remote write failed: read message type: read tcp XXX.XX.11.56:52221->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:04:21 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:27 remote write failed: read message type: read tcp XXX.XX.11.56:52225->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:04:31 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:33 remote write failed: read message type: read tcp XXX.XX.11.56:52228->XXX.XX.11.54:8088: i/o timeout\r\n[handoff] 2015/10/26 18:04:39 remote write failed: read message type: read tcp XXX.XX.11.56:52230->XXX.XX.11.54:8088: i/o timeout\r\n[monitor] 2015/10/26 18:04:41 failed to store statistics: timeout\r\n[handoff] 2015/10/26 18:04:45 remote write failed: read message type: read tcp XXX.XX.11.56:52233->XXX.XX.11.54:8088: i/o timeout"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/151263030","html_url":"https://github.com/influxdb/influxdb/issues/4573#issuecomment-151263030","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4573","id":151263030,"user":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T19:48:22Z","updated_at":"2015-10-26T19:48:22Z","body":"Sounds like one of the WAL files had a partial write. Maybe the server got killed in the middle of a write? Removing the WAL files will let you bring it back up, but it should recover from a corrupt WAL file without failing out.\r\n\r\nWhat does the directory for shard 5 look like?"}}