{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2396","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2396/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2396/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2396/events","html_url":"https://github.com/coreos/etcd/issues/2396","id":59249722,"number":2396,"title":"Error: \"invalid memory address or nil pointer dereference\" when trying to add new node to existing cluster","user":{"login":"wsimmonds","id":2374056,"avatar_url":"https://avatars.githubusercontent.com/u/2374056?v=3","gravatar_id":"","url":"https://api.github.com/users/wsimmonds","html_url":"https://github.com/wsimmonds","followers_url":"https://api.github.com/users/wsimmonds/followers","following_url":"https://api.github.com/users/wsimmonds/following{/other_user}","gists_url":"https://api.github.com/users/wsimmonds/gists{/gist_id}","starred_url":"https://api.github.com/users/wsimmonds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wsimmonds/subscriptions","organizations_url":"https://api.github.com/users/wsimmonds/orgs","repos_url":"https://api.github.com/users/wsimmonds/repos","events_url":"https://api.github.com/users/wsimmonds/events{/privacy}","received_events_url":"https://api.github.com/users/wsimmonds/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/etcd/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-02-27T15:22:35Z","updated_at":"2015-03-02T19:55:03Z","closed_at":null,"body":"Hi,\r\n\r\nI'm currently experiencing an issue when trying to add a new node to an existing cluster. I'm running 2.0.3 from within a Docker container. My log looks somewhat like the following:\r\n\r\n\r\nFeb 27 14:52:58 ip-10-21-1-94.ec2.internal systemd[1]: Starting etcd container...\r\nFeb 27 14:52:58 ip-10-21-1-94.ec2.internal writer_process[4938]: etcd\r\nFeb 27 14:52:58 ip-10-21-1-94.ec2.internal writer_process[4945]: etcd\r\nFeb 27 14:52:58 ip-10-21-1-94.ec2.internal writer_process[4953]: Pulling repository quay.io/coreos/etcd\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal writer_process[4953]: Status: Image is up to date for quay.io/coreos/etcd:v2.0.3\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal systemd[1]: Started etcd container.\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:52:59 etcd: listening for peers on http://0.0.0.0:2380\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:52:59 etcd: listening for peers on http://0.0.0.0:7001\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:52:59 etcd: listening for client requests on http://0.0.0.0:2379\r\nFeb 27 14:52:59 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:52:59 etcd: listening for client requests on http://0.0.0.0:4001\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: could not get cluster response from http://10.21.1.43:2380: Get http://10.21.1.43:2380/members: net/http: request canceled while waiting for connection\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: name = ip-10-21-1-94.ec2.internal\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: data dir = /opt/etcd\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: member dir = /opt/etcd/member\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: heartbeat = 100ms\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: election = 1000ms\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: snapshot count = 10000\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: 2015/02/27 14:53:00 etcdserver: advertise client URLs = http://10.21.1.94:2379,http://10.21.1.94:4001\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: panic: runtime error: invalid memory address or nil pointer dereference\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: [signal 0xb code=0x1 addr=0x0 pc=0x4a5b5d]\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 16 [running]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: runtime.panic(0x7dc1e0, 0xa63493)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/etcdserver.startNode(0xc20803a580, 0x0, 0x0, 0x0, 0xc208044f80, 0x0, 0x0, 0x10, 0x10)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/raft.go:78 +0xdd\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/etcdserver.NewServer(0xc20803a580, 0xc20803a580, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/server.go:171 +0x6bc\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/etcdmain.startEtcd(0xc20807c000, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/etcdmain/etcd.go:162 +0x1013\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/etcdmain.Main()\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/etcdmain/etcd.go:58 +0x1c5\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: main.main()\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/main.go:48 +0x1ab\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 19 [finalizer wait]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: runtime.park(0x4131d0, 0xa67fd8, 0xa66209)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: runtime.parkunlock(0xa67fd8, 0xa66209)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: runfinq()\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: runtime.goexit()\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/proc.c:1445\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 20 [syscall]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: os/signal.loop()\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: created by os/signal.init·1\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x32\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 28 [IO wait]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net.runtime_pollWait(0x7fbec1e58938, 0x72, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net.(*pollDesc).Wait(0xc20807efb0, 0x72, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net.(*pollDesc).WaitRead(0xc20807efb0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net.(*netFD).Read(0xc20807ef50, 0xc208074000, 0x1000, 0x1000, 0x0, 0x7fbec1e573f0, 0xb)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/fd_unix.go:242 +0x34c\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net.(*conn).Read(0xc2080380a0, 0xc208074000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/net.go:122 +0xe7\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x7fbec1e58f30, 0xc2080380a0, 0x12a05f200, 0x12a05f200, 0xc208074000, 0x1000, 0x1000, 0x7fbec1e590b8, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /Users/unihorn/gopath/src/github.com/coreos/etcd/scripts/release/etcd/gopath/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x16f\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc2080405e0, 0xc208074000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: <autogenerated>:21 +0xcc\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.noteEOFReader.Read(0x7fbec1e59068, 0xc2080405e0, 0xc20803a738, 0xc208074000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:1203 +0x72\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.(*noteEOFReader).Read(0xc208040600, 0xc208074000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: <autogenerated>:124 +0xca\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: bufio.(*Reader).fill(0xc2080049c0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/bufio/bufio.go:97 +0x1b3\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: bufio.(*Reader).Peek(0xc2080049c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/bufio/bufio.go:132 +0x101\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.(*persistConn).readLoop(0xc20803a6e0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:782 +0x95\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: created by net/http.(*Transport).dialConn\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:600 +0x93f\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 26 [runnable]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.(*persistConn).readLoop(0xc20803a2c0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:868 +0x829\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: created by net/http.(*Transport).dialConn\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:600 +0x93f\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 27 [select]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.(*persistConn).writeLoop(0xc20803a2c0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:885 +0x38f\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: created by net/http.(*Transport).dialConn\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:601 +0x957\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: goroutine 29 [select]:\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: net/http.(*persistConn).writeLoop(0xc20803a6e0)\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:885 +0x38f\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: created by net/http.(*Transport).dialConn\r\nFeb 27 14:53:00 ip-10-21-1-94.ec2.internal writer_process[4961]: /usr/local/go/src/pkg/net/http/transport.go:601 +0x957\r\nFeb 27 14:53:01 ip-10-21-1-94.ec2.internal systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nFeb 27 14:53:01 ip-10-21-1-94.ec2.internal writer_process[5026]: etcd\r\nFeb 27 14:53:01 ip-10-21-1-94.ec2.internal systemd[1]: Unit etcd.service entered failed state.\r\n\r\n\r\nThe existing cluster consists of 2 nodes, my etcd.service looks like this:\r\n\r\n\r\n[Service]\r\nEnvironmentFile=/etc/environment\r\nTimeoutStartSec=0\r\nSyslogIdentifier=writer_process\r\nExecStartPre=-/usr/bin/docker kill etcd\r\nExecStartPre=-/usr/bin/docker rm etcd\r\nExecStartPre=/usr/bin/docker pull quay.io/coreos/etcd:v2.0.3\r\nExecStart=/bin/bash -c \"/usr/bin/docker run \\\r\n  -p 2379:2379 \\\r\n  -p 2380:2380 \\\r\n  -p 4001:4001 \\\r\n  -p 7001:7001 \\\r\n  --name etcd \\\r\n  -v /opt/etcd:/opt/etcd \\\r\n  -v /usr/share/ca-certificates/:/etc/ssl/certs \\\r\n  quay.io/coreos/etcd:v2.0.3 \\\r\n  -data-dir /opt/etcd \\\r\n  -name %H \\\r\n  -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \\\r\n  -advertise-client-urls http://10.21.1.94:2379,http://10.21.1.94:4001 \\\r\n  -listen-peer-urls http://0.0.0.0:2380,http://0.0.0.0:7001 \\\r\n  -initial-advertise-peer-urls http://10.21.1.94:2380,http://10.21.1.94:7001 \\\r\n  -initial-cluster-state existing \\\r\n  -initial-cluster ip-10-21-1-43.ec2.internal=http://10.21.1.43:2380,ip-10-21-1-43.ec2.internal=http://10.21.1.43:7001,ip-10-21-2-166.ec2.internal=http://10.21.2.166:2380,ip-10-21-2-166.ec2.internal=http://10.21.2.166:7001\"\r\nExecStop=/usr/bin/docker kill etcd\r\n\r\nI thought at first it might be related to the \"could not get cluster response\", but a quick check from .94 works:\r\n\r\ncore@ip-10-21-1-94 /etc/systemd/system $ curl -L http://10.21.1.43:2380/members\r\n[{\"id\":12394471189341718283,\"peerURLs\":[\"http://10.21.2.166:2380\",\"http://10.21.2.166:7001\"],\"name\":\"ip-10-21-2-166.ec2.internal\",\"clientURLs\":[\"http://10.21.2.166:2379\",\"http://10.21.2.166:4001\"]},{\"id\":13661735980988760828,\"peerURLs\":[\"http://10.21.1.43:2380\",\"http://10.21.1.43:7001\"],\"name\":\"ip-10-21-1-43.ec2.internal\",\"clientURLs\":[\"http://10.21.1.43:2379\",\"http://10.21.1.43:4001\"]}]\r\n\r\nAny help would be hugely appreciated."},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/76799201","html_url":"https://github.com/coreos/etcd/issues/2396#issuecomment-76799201","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2396","id":76799201,"user":{"login":"timhaak","id":271607,"avatar_url":"https://avatars.githubusercontent.com/u/271607?v=3","gravatar_id":"","url":"https://api.github.com/users/timhaak","html_url":"https://github.com/timhaak","followers_url":"https://api.github.com/users/timhaak/followers","following_url":"https://api.github.com/users/timhaak/following{/other_user}","gists_url":"https://api.github.com/users/timhaak/gists{/gist_id}","starred_url":"https://api.github.com/users/timhaak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timhaak/subscriptions","organizations_url":"https://api.github.com/users/timhaak/orgs","repos_url":"https://api.github.com/users/timhaak/repos","events_url":"https://api.github.com/users/timhaak/events{/privacy}","received_events_url":"https://api.github.com/users/timhaak/received_events","type":"User","site_admin":false},"created_at":"2015-03-02T19:55:02Z","updated_at":"2015-03-02T19:55:02Z","body":"Yes think I'm doing something wrong. Was more that I assume it should say x wrong not give run time error. \r\n\r\nSorted out what I needed on my side so not critical to me :) \r\n\r\nI should have been using proxy not existing. Was a misunderstanding of the model on my side. I assumed you bootstrapped then added all the servers and they decide between themselves which were active and which slave (proxy). "}}