{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2050","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2050/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2050/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2050/events","html_url":"https://github.com/coreos/etcd/issues/2050","id":53590869,"number":2050,"title":"etcd 0.4.6 panics on new members when data directory is on a different mount","user":{"login":"carmstrong","id":536009,"avatar_url":"https://avatars.githubusercontent.com/u/536009?v=3","gravatar_id":"","url":"https://api.github.com/users/carmstrong","html_url":"https://github.com/carmstrong","followers_url":"https://api.github.com/users/carmstrong/followers","following_url":"https://api.github.com/users/carmstrong/following{/other_user}","gists_url":"https://api.github.com/users/carmstrong/gists{/gist_id}","starred_url":"https://api.github.com/users/carmstrong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/carmstrong/subscriptions","organizations_url":"https://api.github.com/users/carmstrong/orgs","repos_url":"https://api.github.com/users/carmstrong/repos","events_url":"https://api.github.com/users/carmstrong/events{/privacy}","received_events_url":"https://api.github.com/users/carmstrong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-01-07T02:38:17Z","updated_at":"2015-01-08T23:59:06Z","closed_at":null,"body":"We wanted to move the etcd data directory to EC2's ephemeral disk. I formatted the volume as ext4, mounted it, and configured etcd to use it. However, when a node joins the cluster, etcd panics. I tried both ext3 and ext4 (and btrfs) with the same result.\r\n\r\n```\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: [etcd] Jan  7 01:11:19.169 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: peer added: '79aa20eea1db4e469770c2698a55c03d'\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: panic: runtime error: invalid memory address or nil pointer dereference\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: [signal 0xb code=0x1 addr=0x0 pc=0x56e1cf]\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: goroutine 10813 [running]:\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: runtime.panic(0x897820, 0xcb21f3)\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: github.com/coreos/etcd/third_party/github.com/goraft/raft.(*Peer).flush(0xc208ce9c00)\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /build/amd64-usr/var/tmp/portage/dev-db/etcd-0.4.6-r2/work/etcd-0.4.6/gopath/src/github.com/coreos/etcd/third_party/github.com/goraft/raft/peer.go:180 +0x2af\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: github.com/coreos/etcd/third_party/github.com/goraft/raft.(*Peer).heartbeat(0xc208ce9c00, 0xc20a278660)\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /build/amd64-usr/var/tmp/portage/dev-db/etcd-0.4.6-r2/work/etcd-0.4.6/gopath/src/github.com/coreos/etcd/third_party/github.com/goraft/raft/peer.go:163 +0x48a\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: github.com/coreos/etcd/third_party/github.com/goraft/raft.funcÂ·005()\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /build/amd64-usr/var/tmp/portage/dev-db/etcd-0.4.6-r2/work/etcd-0.4.6/gopath/src/github.com/coreos/etcd/third_party/github.com/goraft/raft/peer.go:97 +0x76\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: created by github.com/coreos/etcd/third_party/github.com/goraft/raft.(*Peer).startHeartbeat\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /build/amd64-usr/var/tmp/portage/dev-db/etcd-0.4.6-r2/work/etcd-0.4.6/gopath/src/github.com/coreos/etcd/third_party/github.com/goraft/raft/peer.go:98 +0x13d\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: goroutine 16 [select, 41 minutes]:\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: github.com/coreos/etcd/etcd.(*Etcd).runServer(0xc208060300)\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: /build/amd64-usr/var/tmp/portage/dev-db/etcd-0.4.6-r2/work/etcd-0.4.6/gopath/src/github.com/coreos/etcd/etcd/etcd.go:327 +0xa4a\r\nJan 07 01:11:19 ip-10-21-2-54.ec2.internal etcd[609]: github.com/coreos/etcd/etcd.(*Etcd).Run(0xc208060300)\r\n<< snipped >>\r\nJan 07 01:11:29 ip-10-21-2-54.ec2.internal systemd[1]: etcd.service holdoff time over, scheduling restart.\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal systemd[1]: Stopping etcd...\r\n```\r\n\r\nFor the full stacktrace: https://gist.githubusercontent.com/carmstrong/fd274a839234b63b8d6f/raw/1c067f9fbdd23e51df27e451ea725ea6dd085589/gistfile1.txt\r\n\r\nNote that etcd then gets confused:\r\n```\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal systemd[1]: Starting etcd...\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal systemd[1]: Started etcd.\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:30.740 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: peer added: '79aa20eea1db4e469770c2698a55c03d'\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:30.741 INFO      | Peer URLs in log:  / 66239884f41d4adeb0ea4a106c38e9fb (http://10.21.1.245:7001)\r\nJan 07 01:11:30 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:30.741 INFO      | Discovery via https://discovery.etcd.io using prefix /ccf1ce2060eefa4c1919ab1b132fe38a.\r\nJan 07 01:11:31 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:31.351 INFO      | Discovery found peers [http://10.21.1.83:7001 http://10.21.1.82:7001 http://10.21.1.245:7001]\r\nJan 07 01:11:31 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:31.351 INFO      | Discovery fetched back peer list: [10.21.1.83:7001 10.21.1.82:7001 10.21.1.245:7001]\r\nJan 07 01:11:31 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:31.356 INFO      | Send Join Request to http://10.21.1.245:7001/join\r\nJan 07 01:11:31 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:31.361 INFO      | 66239884f41d4adeb0ea4a106c38e9fb attempted to join via 10.21.1.245:7001 failed: fail on join request: Raft Internal Error ()\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.862 INFO      | 66239884f41d4adeb0ea4a106c38e9fb attempted to join via 10.21.1.83:7001 failed: fail checking join version: Client Internal Error (Get http://10.21.1.83:7001/version: dial tcp 10.21.1.83:7001: i/o timeout)\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.867 INFO      | Send Join Request to http://10.21.1.82:7001/join\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.869 INFO      | 66239884f41d4adeb0ea4a106c38e9fb attempted to join via 10.21.1.82:7001 failed: fail on join request: Raft Internal Error ()\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.875 INFO      | Send Join Request to http://10.21.1.245:7001/join\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.877 INFO      | 66239884f41d4adeb0ea4a106c38e9fb attempted to join via 10.21.1.245:7001 failed: fail on join request: Raft Internal Error ()\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.877 WARNING   | 66239884f41d4adeb0ea4a106c38e9fb cannot connect to previous cluster [10.21.1.245:7001 10.21.1.83:7001 10.21.1.82:7001 10.21.1.245:7001]: fail joining the cluster via given peers after 1 retries\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.879 INFO      | etcd server [name 66239884f41d4adeb0ea4a106c38e9fb, listen on :4001, advertised url http://10.21.2.54:4001]\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.879 INFO      | peer server [name 66239884f41d4adeb0ea4a106c38e9fb, listen on :7001, advertised url http://10.21.2.54:7001]\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.879 INFO      | 66239884f41d4adeb0ea4a106c38e9fb starting in peer mode\r\nJan 07 01:11:35 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:35.879 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: state changed from 'initialized' to 'follower'.\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.320 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: state changed from 'follower' to 'candidate'.\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.322 INFO      | removed during cluster re-configuration\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.322 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: state changed from 'candidate' to 'follower'.\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.322 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: term #18446744073709551615 started.\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.322 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: state changed from 'follower' to 'stopped'.\r\nJan 07 01:11:38 ip-10-21-2-54.ec2.internal etcd[9261]: [etcd] Jan  7 01:11:38.323 INFO      | 66239884f41d4adeb0ea4a106c38e9fb: state changed from 'stopped' to 'stopped'.\r\n```\r\n\r\nDoes anyone have any ideas?"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/69272621","html_url":"https://github.com/coreos/etcd/issues/2050#issuecomment-69272621","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2050","id":69272621,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-01-08T23:59:06Z","updated_at":"2015-01-08T23:59:06Z","body":"@carmstrong We cannot reproduce this problem. Here is what we did.\r\n\r\n1. Starts a ec2 m3.large MachineA. And starts an 0.4.6 etcd process on the local filesystem (/dev/xdva1).\r\n2. push some traffic to the etcd cluster to make it save a snapshot. \r\n3. stop the etcd process.\r\n4. attach a 8Gi SSD volume to Machine A. \r\n5. mkfs -t ext4 on the new volume\r\n6. mount the volume onto /media/ephemeral\r\n7. copy the previous data dir to /media/ephemeral\r\n8. change the ownership of the data dir to the current user.\r\n9. restart etcd using the moved data dir.\r\n(10) push some traffic to etcd to make it save a snapshot (we also tried to skip this step)  \r\n11. Starts a ec2 m3.large Machine B. And let Machine B join Machine A. \r\n\r\nIt works fine. Did we miss anything? "}}