{"action":"closed","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4545","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4545/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4545/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4545/events","html_url":"https://github.com/influxdb/influxdb/issues/4545","id":112846763,"number":4545,"title":"tsm1 checkpoint recovery error","user":{"login":"benbjohnson","id":118015,"avatar_url":"https://avatars.githubusercontent.com/u/118015?v=3","gravatar_id":"","url":"https://api.github.com/users/benbjohnson","html_url":"https://github.com/benbjohnson","followers_url":"https://api.github.com/users/benbjohnson/followers","following_url":"https://api.github.com/users/benbjohnson/following{/other_user}","gists_url":"https://api.github.com/users/benbjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/benbjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbjohnson/subscriptions","organizations_url":"https://api.github.com/users/benbjohnson/orgs","repos_url":"https://api.github.com/users/benbjohnson/repos","events_url":"https://api.github.com/users/benbjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/benbjohnson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/2%20-%20Working","name":"2 - Working","color":"CCCCCC"},{"url":"https://api.github.com/repos/influxdb/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"closed","locked":false,"assignee":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"milestone":null,"comments":0,"created_at":"2015-10-22T16:58:15Z","updated_at":"2015-10-23T18:32:58Z","closed_at":"2015-10-23T18:32:58Z","body":"While running the stress tool, the server hung indefinitely on this lock:\r\n\r\nhttps://github.com/influxdb/influxdb/blob/master/tsdb/engine/tsm1/tsm1.go#L2003\r\n\r\nI killed the server via `SIGQUIT` and then started it back up and it's unable to start because of this panic:\r\n\r\n```\r\npanic: error removing checkpoint file: remove /Users/benbjohnson/.influxdb/data/stress/default/2/0000046.tsm1.check: no such file or directory\r\n\r\ngoroutine 1 [running]:\r\ngithub.com/influxdb/influxdb/tsdb/engine/tsm1.(*compactionCheckpoint).cleanup(0xc82025bf80)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:2131 +0x7a1\r\ngithub.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).cleanupUnfinishedCompaction(0xc820229440)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:299 +0x8ea\r\ngithub.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).Open(0xc820229440, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:227 +0x148\r\ngithub.com/influxdb/influxdb/tsdb.(*Shard).Open.func1(0xc820272c60, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/shard.go:123 +0x365\r\ngithub.com/influxdb/influxdb/tsdb.(*Shard).Open(0xc820272c60, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/shard.go:100 +0x2d\r\ngithub.com/influxdb/influxdb/tsdb.(*Store).loadShards(0xc820158000, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xd2c\r\ngithub.com/influxdb/influxdb/tsdb.(*Store).Open(0xc820158000, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tsdb/store.go:371 +0x35a\r\ngithub.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1(0xc820134360, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:392 +0xe08\r\ngithub.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open(0xc820134360, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:338 +0x2d\r\ngithub.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc82014e000, 0xc820068180, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:121 +0xe27\r\nmain.(*Main).Run(0xc820111f00, 0xc820068180, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:86 +0x6a1\r\nmain.main()\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:46 +0x29b\r\n\r\ngoroutine 17 [syscall, locked to thread]:\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1\r\n\r\ngoroutine 19 [syscall]:\r\nos/signal.loop()\r\n\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x18\r\ncreated by os/signal.init.1\r\n\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x37\r\n\r\ngoroutine 35 [IO wait]:\r\nnet.runtime_pollWait(0xfc1f50, 0x72, 0xc8200680a0)\r\n\t/usr/local/go/src/runtime/netpoll.go:157 +0x60\r\nnet.(*pollDesc).Wait(0xc82014a290, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitRead(0xc82014a290, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r\nnet.(*netFD).accept(0xc82014a230, 0x0, 0xf40028, 0xc820010240)\r\n\t/usr/local/go/src/net/fd_unix.go:408 +0x27c\r\nnet.(*TCPListener).AcceptTCP(0xc820126088, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d\r\nnet.(*TCPListener).Accept(0xc820126088, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d\r\ngithub.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc820122fc0, 0xfc0fd0, 0xc820126088, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tcp/mux.go:48 +0x4b\r\ncreated by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:376 +0xafb\r\n\r\ngoroutine 36 [chan receive]:\r\ngithub.com/influxdb/influxdb/tcp.(*listener).Accept(0xc820126090, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f\r\ngithub.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc82015a9c0, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:2129 +0x51\r\ngithub.com/hashicorp/raft.(*NetworkTransport).listen(0xc82014a310)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/net_transport.go:346 +0x79\r\ncreated by github.com/hashicorp/raft.NewNetworkTransport\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/net_transport.go:138 +0x284\r\n\r\ngoroutine 37 [select]:\r\ngithub.com/hashicorp/raft.(*Raft).leaderLoop(0xc820178000)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:826 +0xcce\r\ngithub.com/hashicorp/raft.(*Raft).runLeader(0xc820178000)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:798 +0x88e\r\ngithub.com/hashicorp/raft.(*Raft).run(0xc820178000)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:595 +0xbb\r\ngithub.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:245 +0x20\r\ngithub.com/hashicorp/raft.(*raftState).goFunc.func1(0xc820178000, 0xc820155be0)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:152 +0x44\r\ncreated by github.com/hashicorp/raft.(*raftState).goFunc\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:153 +0x4d\r\n\r\ngoroutine 38 [select]:\r\ngithub.com/hashicorp/raft.(*Raft).runFSM(0xc820178000)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:502 +0x10eb\r\ngithub.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runFSM)-fm()\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:246 +0x20\r\ngithub.com/hashicorp/raft.(*raftState).goFunc.func1(0xc820178000, 0xc820155bf0)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:152 +0x44\r\ncreated by github.com/hashicorp/raft.(*raftState).goFunc\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:153 +0x4d\r\n\r\ngoroutine 39 [select]:\r\ngithub.com/hashicorp/raft.(*Raft).runSnapshots(0xc820178000)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:1613 +0x493\r\ngithub.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runSnapshots)-fm()\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/raft.go:247 +0x20\r\ngithub.com/hashicorp/raft.(*raftState).goFunc.func1(0xc820178000, 0xc820155c00)\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:152 +0x44\r\ncreated by github.com/hashicorp/raft.(*raftState).goFunc\r\n\t/Users/benbjohnson/go/src/github.com/hashicorp/raft/state.go:153 +0x4d\r\n\r\ngoroutine 40 [select]:\r\ngithub.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0xc82011cc80)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/state.go:189 +0x4f5\r\ncreated by github.com/influxdb/influxdb/meta.(*localRaft).open\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/state.go:179 +0x1344\r\n\r\ngoroutine 41 [chan receive]:\r\ngithub.com/influxdb/influxdb/tcp.(*listener).Accept(0xc820126098, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f\r\ngithub.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc820158140)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:622 +0xa2\r\ncreated by github.com/influxdb/influxdb/meta.(*Store).Open\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:239 +0x364\r\n\r\ngoroutine 42 [chan receive]:\r\ngithub.com/influxdb/influxdb/tcp.(*listener).Accept(0xc8201260a0, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f\r\ngithub.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0xc820158140)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:726 +0xa2\r\ncreated by github.com/influxdb/influxdb/meta.(*Store).Open\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:242 +0x3b6\r\n\r\ngoroutine 5 [select]:\r\ngithub.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc820134360, 0xc82012e4e0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:531 +0x145\r\ncreated by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:382 +0xc7f\r\n\r\ngoroutine 6 [select]:\r\ngithub.com/influxdb/influxdb/meta.(*Store).WaitForLeader(0xc820158140, 0x6fc23ac00, 0x0, 0x0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/meta/store.go:556 +0x3fc\r\ngithub.com/influxdb/influxdb/monitor.(*Monitor).storeStatistics(0xc8201480e0)\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/monitor/service.go:341 +0x2dd\r\ncreated by github.com/influxdb/influxdb/monitor.(*Monitor).Open\r\n\t/Users/benbjohnson/go/src/github.com/influxdb/influxdb/monitor/service.go:138 +0x349\r\n```\r\n\r\nHere's the contents of that directory:\r\n\r\n```\r\n$ ls -al ~/.influxdb/data/stress/default/2\r\ntotal 549952\r\ndrwxr-xr-x  23 benbjohnson  staff        782 Oct 22 10:48 .\r\ndrwx------   4 benbjohnson  staff        136 Oct 22 10:30 ..\r\n-rw-r--r--   1 benbjohnson  staff    4804269 Oct 22 10:41 0000044.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    5754566 Oct 22 10:41 0000045.tsm1\r\n-rw-r--r--   1 benbjohnson  staff        561 Oct 22 10:48 0000046.compact\r\n-rw-r--r--   1 benbjohnson  staff   92472278 Oct 22 10:44 0000046.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4986479 Oct 22 10:42 0000047.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4174146 Oct 22 10:42 0000050.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4138344 Oct 22 10:43 0000053.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4888084 Oct 22 10:44 0000054.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    5397278 Oct 22 10:45 0000058.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    5141573 Oct 22 10:45 0000060.tsm1\r\n-rw-r--r--   1 benbjohnson  staff  120201530 Oct 22 10:48 0000061.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    3735902 Oct 22 10:45 0000062.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4920712 Oct 22 10:46 0000063.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    1901348 Oct 22 10:46 0000065.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4716518 Oct 22 10:46 0000066.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    5528751 Oct 22 10:47 0000068.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    3517831 Oct 22 10:47 0000069.tsm1\r\n-rw-r--r--   1 benbjohnson  staff    4738569 Oct 22 10:47 0000070.tsm1\r\n-rw-r--r--   1 benbjohnson  staff         69 Oct 22 10:30 fields\r\n-rw-r--r--   1 benbjohnson  staff     280430 Oct 22 10:30 ids\r\n-rw-r--r--   1 benbjohnson  staff     227863 Oct 22 10:30 series\r\n```"}}