{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2651","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2651/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2651/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2651/events","html_url":"https://github.com/coreos/etcd/issues/2651","id":67530081,"number":2651,"title":"etcd panic when using etcdctl remove a member","user":{"login":"holmes86","id":1610303,"avatar_url":"https://avatars.githubusercontent.com/u/1610303?v=3","gravatar_id":"","url":"https://api.github.com/users/holmes86","html_url":"https://github.com/holmes86","followers_url":"https://api.github.com/users/holmes86/followers","following_url":"https://api.github.com/users/holmes86/following{/other_user}","gists_url":"https://api.github.com/users/holmes86/gists{/gist_id}","starred_url":"https://api.github.com/users/holmes86/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holmes86/subscriptions","organizations_url":"https://api.github.com/users/holmes86/orgs","repos_url":"https://api.github.com/users/holmes86/repos","events_url":"https://api.github.com/users/holmes86/events{/privacy}","received_events_url":"https://api.github.com/users/holmes86/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-04-10T07:18:30Z","updated_at":"2015-06-02T22:45:18Z","closed_at":null,"body":"I try removing a member of etcd cluster, then I found etcd panic. \"journalctl -u etcd\" as follow:\r\n\r\n```\r\nApr 10 15:07:01 machine2 systemd[1]: Unit etcd.service entered failed state.\r\nApr 10 15:07:01 machine2 systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/http/server.go:1721 +0x313\r\nApr 10 15:07:01 machine2 etcd[13510]: created by net/http.(*Server).Serve\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/http/server.go:1174 +0xa7e\r\nApr 10 15:07:01 machine2 etcd[13510]: net/http.(*conn).serve(0xc208314000)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/http/server.go:1673 +0x19f\r\nApr 10 15:07:01 machine2 etcd[13510]: net/http.serverHandler.ServeHTTP(0xc2081e3e60, 0x7f48a0634d30, 0xc208436aa0, 0xc20807fc70)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/cors/cors.go:86 +0x20f\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/pkg/cors.(*CORSHandler).ServeHTTP(0xc20833c7e0, 0x7f48a0634d30, 0xc208436aa0, 0xc20807fc70)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/http/server.go:1511 +0x1a3\r\nApr 10 15:07:01 machine2 etcd[13510]: net/http.(*ServeMux).ServeHTTP(0xc208021dd0, 0x7f48a0634d30, 0xc208436aa0, 0xc20807fc70)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdserver/etcdhttp/client.go:116 +0x38f\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdserver/etcdhttp.(*keysHandler).ServeHTTP(0xc2084ff740, 0x7f48a0634d30, 0xc208436aa0, 0xc20807fc70)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdserver/server.go:491 +0x5cf\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdserver.(*EtcdServer).Do(0xc208088100, 0x7f48a0634b70, 0xc20818e0c0, 0xb94ca227363d0002, 0xc2083cf8f0, 0x3, 0xc2083d3e50, 0x45, 0xc2080a1b00, 0x8c, ...)\r\nApr 10 15:07:01 machine2 etcd[13510]: goroutine 36 [runnable]:\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/fileutil/purge.go:75 +0x157\r\nApr 10 15:07:01 machine2 etcd[13510]: created by github.com/coreos/etcd/pkg/fileutil.PurgeFile\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/fileutil/purge.go:69 +0x5e5\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/pkg/fileutil.func·001()\r\nApr 10 15:07:01 machine2 etcd[13510]: goroutine 35 [runnable]:\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/osutil/osutil.go:82 +0x147\r\nApr 10 15:07:01 machine2 etcd[13510]: created by github.com/coreos/etcd/pkg/osutil.HandleInterrupts\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/osutil/osutil.go:66 +0x5e\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/pkg/osutil.func·001()\r\nApr 10 15:07:01 machine2 etcd[13510]: goroutine 34 [chan receive]:\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/etcd.go:189 +0x14a4\r\nApr 10 15:07:01 machine2 etcd[13510]: created by github.com/coreos/etcd/etcdmain.startEtcd\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/etcd.go:188 +0x81\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdmain.func·006(0x7f48a0625c38, 0xc208001230)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/http.go:36 +0x136\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdmain.serveHTTP(0x7f48a0625c38, 0xc208001230, 0x7f48a0634bb0, 0xc20833c7e0, 0x0, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/http/server.go:1698 +0x91\r\nApr 10 15:07:01 machine2 etcd[13510]: net/http.(*Server).Serve(0xc2081e3e60, 0x7f48a0625c38, 0xc208001230, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:48 +0x70\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept(0xc208001230, 0x0, 0x0, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/tcpsock_posix.go:244 +0x4b\r\nApr 10 15:07:01 machine2 etcd[13510]: net.(*TCPListener).Accept(0xc208048030, 0x0, 0x0, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/tcpsock_posix.go:234 +0x5d\r\nApr 10 15:07:01 machine2 etcd[13510]: net.(*TCPListener).AcceptTCP(0xc208048030, 0x1d, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/fd_unix.go:419 +0x343\r\nApr 10 15:07:01 machine2 etcd[13510]: net.(*netFD).accept(0xc20808c850, 0x8f6fa8, 0x0, 0x7f48a06243f0, 0xb)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/fd_poll_runtime.go:89 +0x42\r\nApr 10 15:07:01 machine2 etcd[13510]: net.(*pollDesc).WaitRead(0xc20808c8b0, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/net/fd_poll_runtime.go:84 +0x46\r\nApr 10 15:07:01 machine2 etcd[13510]: net.(*pollDesc).Wait(0xc20808c8b0, 0x72, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /builddir/build/BUILD/go/src/pkg/runtime/netpoll.goc:146 +0x66\r\nApr 10 15:07:01 machine2 etcd[13510]: net.runtime_pollWait(0x7f48a0625940, 0x72, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: goroutine 33 [IO wait]:\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/etcd.go:189 +0x14a4\r\nApr 10 15:07:01 machine2 etcd[13510]: created by github.com/coreos/etcd/etcdmain.startEtcd\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/etcd.go:188 +0x81\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdmain.func·006(0x7f48a0625c38, 0xc2080011c0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdmain/http.go:36 +0x136\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdmain.serveHTTP(0x7f48a0625c38, 0xc2080011c0, 0x7f48a0634bb0, 0xc20833c7e0, 0x0, 0x0, 0x0)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdserver/server.go:733 +0x9ba\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdserver.(*EtcdServer).applyRequest(0xc208088100, 0x964ca225fa210001, 0xc2084e7cd0, 0x3, 0xc2084f1c50, 0x25, 0xc2084f3b60, 0x51, 0x0, 0x0, ...)\r\nApr 10 15:07:01 machine2 etcd[13510]: /root/rpmbuild/BUILD/etcd-2.0.3/src/github.com/coreos/etcd/etcdserver/cluster.go:331 +0xb1\r\nApr 10 15:07:01 machine2 etcd[13510]: github.com/coreos/etcd/etcdserver.(*Cluster).UpdateAttributes(0xc208352140, 0x6fafd2d770d2396, 0xc2084e7cf0, 0xb, 0xc2084345c0, 0x2, 0x4)\r\nApr 10 15:07:01 machine2 etcd[13510]: /usr/lib/golang/src/pkg/runtime/panic.c:279 +0xf5\r\nApr 10 15:07:01 machine2 etcd[13510]: runtime.panic(0x7d9900, 0xa5c5d3)\r\nApr 10 15:07:01 machine2 etcd[13510]: goroutine 27 [running]:\r\nApr 10 15:07:01 machine2 etcd[13510]: [signal 0xb code=0x1 addr=0x0 pc=0x4a11b1]\r\nApr 10 15:07:01 machine2 etcd[13510]: panic: runtime error: invalid memory address or nil pointer dereference\r\nApr 10 15:07:01 machine2 etcd[13510]: 2015/04/10 15:07:01 etcdserver: removed member 6fafd2d770d2396 from cluster 5a21426bca2d472c\r\nApr 10 15:07:01 machine2 etcd[13510]: 2015/04/10 15:07:01 raft: newRaft f2882b486b73afb9 [peers: [6fafd2d770d2396,f2882b486b73afb9], term: 6113, commit: 12823270, applied: 12819995, lastindex: 12823270, lastterm: 6113]\r\nApr 10 15:07:01 machine2 etcd[13510]: 2015/04/10 15:07:01 raft: f2882b486b73afb9 became follower at term 6113\r\nApr 10 15:07:01 machine2 etcd[13510]: 2015/04/10 15:07:01 etcdserver: restart member f2882b486b73afb9 in cluster 5a21426bca2d472c at commit index 12823270\r\n\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/108120268","html_url":"https://github.com/coreos/etcd/issues/2651#issuecomment-108120268","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2651","id":108120268,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-06-02T22:45:18Z","updated_at":"2015-06-02T22:45:18Z","body":"@holmes86 Any update?"}}