{"action":"closed","issue":{"url":"https://api.github.com/repos/atomix/copycat/issues/24","labels_url":"https://api.github.com/repos/atomix/copycat/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/atomix/copycat/issues/24/comments","events_url":"https://api.github.com/repos/atomix/copycat/issues/24/events","html_url":"https://github.com/atomix/copycat/issues/24","id":112650285,"number":24,"title":"Possible log file corruption when a node is terminated","user":{"login":"madjam","id":8388825,"avatar_url":"https://avatars.githubusercontent.com/u/8388825?v=3","gravatar_id":"","url":"https://api.github.com/users/madjam","html_url":"https://github.com/madjam","followers_url":"https://api.github.com/users/madjam/followers","following_url":"https://api.github.com/users/madjam/following{/other_user}","gists_url":"https://api.github.com/users/madjam/gists{/gist_id}","starred_url":"https://api.github.com/users/madjam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/madjam/subscriptions","organizations_url":"https://api.github.com/users/madjam/orgs","repos_url":"https://api.github.com/users/madjam/repos","events_url":"https://api.github.com/users/madjam/events{/privacy}","received_events_url":"https://api.github.com/users/madjam/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-21T18:25:40Z","updated_at":"2015-10-21T23:41:37Z","closed_at":"2015-10-21T23:41:37Z","body":"I happened to notice this occasionally during some recent runs. Doesn't happen every time.\r\n\r\nHere's a scenario I was running:\r\n- 3 node cluster. s1, s2 and s3.\r\n- A separate client is logging commands on this cluster at the rate of few hundred per second.\r\n- Terminate a follower node (kill java process)\r\n- Try bringing it back up and you see the below exception trace.\r\n\r\n09:04:31.681 [copycat-server-Address[localhost/127.0.0.1:5003]] ERROR i.a.c.u.c.SingleThreadContext - An uncaught exception occurred\r\njava.lang.IndexOutOfBoundsException: null\r\n\tat io.atomix.catalyst.buffer.AbstractBytes.checkOffset(AbstractBytes.java:47) ~[catalyst-buffer-1.0.0-rc3.jar:na]\r\n\tat io.atomix.catalyst.buffer.AbstractBytes.checkRead(AbstractBytes.java:54) ~[catalyst-buffer-1.0.0-rc3.jar:na]\r\n\tat io.atomix.catalyst.buffer.FileBytes.readUnsignedShort(FileBytes.java:286) ~[catalyst-buffer-1.0.0-rc3.jar:na]\r\n\tat io.atomix.catalyst.buffer.AbstractBuffer.readUnsignedShort(AbstractBuffer.java:528) ~[catalyst-buffer-1.0.0-rc3.jar:na]\r\n\tat io.atomix.copycat.server.storage.Segment.<init>(Segment.java:81) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.SegmentManager.loadDiskSegment(SegmentManager.java:345) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.SegmentManager.loadSegment(SegmentManager.java:332) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.SegmentManager.loadSegments(SegmentManager.java:404) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.SegmentManager.open(SegmentManager.java:95) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.SegmentManager.<init>(SegmentManager.java:58) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.Log.<init>(Log.java:135) ~[classes/:na]\r\n\tat io.atomix.copycat.server.storage.Storage.open(Storage.java:280) ~[classes/:na]\r\n\tat io.atomix.copycat.server.state.ServerContext.lambda$0(ServerContext.java:72) ~[classes/:na]\r\n\tat io.atomix.copycat.server.state.ServerContext$$Lambda$3/846238611.run(Unknown Source) ~[na:na]\r\n\tat io.atomix.catalyst.util.concurrent.Runnables.lambda$logFailure$17(Runnables.java:20) ~[catalyst-common-1.0.0-rc3.jar:na]\r\n\tat io.atomix.catalyst.util.concurrent.Runnables$$Lambda$4/1663166483.run(Unknown Source) [catalyst-common-1.0.0-rc3.jar:na]\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_25]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_25]\r\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_25]\r\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_25]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]"}}