{"action":"closed","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/1042","labels_url":"https://api.github.com/repos/coreos/bugs/issues/1042/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/1042/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/1042/events","html_url":"https://github.com/coreos/bugs/issues/1042","id":122625808,"number":1042,"title":"Docker panics following 404 for blob pull","user":{"login":"dan-compton","id":12177003,"avatar_url":"https://avatars.githubusercontent.com/u/12177003?v=3","gravatar_id":"","url":"https://api.github.com/users/dan-compton","html_url":"https://github.com/dan-compton","followers_url":"https://api.github.com/users/dan-compton/followers","following_url":"https://api.github.com/users/dan-compton/following{/other_user}","gists_url":"https://api.github.com/users/dan-compton/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-compton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-compton/subscriptions","organizations_url":"https://api.github.com/users/dan-compton/orgs","repos_url":"https://api.github.com/users/dan-compton/repos","events_url":"https://api.github.com/users/dan-compton/events{/privacy}","received_events_url":"https://api.github.com/users/dan-compton/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-16T23:18:17Z","updated_at":"2015-12-17T00:11:19Z","closed_at":"2015-12-17T00:11:19Z","body":"Docker panics with the stack trace seen below when it encounters a 404 for a blob.  This happens most every time given the following configuration.  \r\n\r\n\r\nWe have 3 systemd units,  nsqd, etcd, and connector listed below:\r\n```\r\n[Unit]\r\nDescription=nsqd\r\nAfter=env-lock.service docker.service\r\nBefore=connector.service\r\nRequires=env-lock.service docker.service\r\n\r\n[Service]\r\nRestart=always\r\nUser=core\r\nTimeoutStartSec=0\r\nRestartSec=5s\r\nExecStartPre=-/usr/bin/docker stop -t 5 %p\r\nExecStartPre=-/usr/bin/docker rm %p\r\nExecStartPre=/usr/bin/docker pull nsqio/nsq:v0.3.5\r\nExecStartPre=/usr/bin/docker run -d --name %p --expose 4150 --expose 4151 nsqio/nsq:v0.3.6 /nsqd\r\nExecStart=/usr/bin/docker logs -f %p\r\nExecStop=/usr/bin/docker stop -t 5 %p\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n```\r\n\r\n```\r\n[Service]\r\nRestart=always\r\nUser=core\r\nTimeoutStartSec=0\r\nRestartSec=5s\r\nExecStartPre=-/usr/bin/docker stop -t 5 %p\r\nExecStartPre=-/usr/bin/docker rm %p\r\nExecStartPre=/usr/bin/docker pull quay.io/coreos/etcd:v2.0.8\r\nExecStartPre=/usr/bin/docker run -d --name %p --expose 2379 quay.io/coreos/etcd:v2.0.8 -name bastion_etcd -advertise-client-urls http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 -initial-advertise-peer-urls http://127.0.0.1:2380 -listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token opsee -initial-cluster bastion_etcd=http://127.0.0.1:2380 -initial-cluster-state new\r\nExecStart=/usr/bin/docker logs -f %p\r\nExecStop=/usr/bin/docker stop -t 5 %p\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n```\r\n\r\n```\r\n[Unit]\r\nDescription=Bastion - Connector\r\nAfter=env-lock.service docker.service bastion-etcd.service nsqd.service bastion-etcd.service\r\nBindsTo=nsqd.service bastion-etcd.service\r\n\r\n[Service]\r\nEnvironmentFile=/etc/opsee/bastion-env.sh\r\nRestart=always\r\nUser=core\r\nTimeoutStartSec=0\r\nRestartSec=5s\r\nEnvironmentFile=/etc/opsee/bastion-env.sh\r\nExecStartPre=-/usr/bin/docker stop -t 5 %p\r\nExecStartPre=-/usr/bin/docker rm %p\r\nExecStartPre=/usr/bin/docker pull quay.io/opsee/gozer:latest\r\nExecStartPre=/usr/bin/docker run -d -p 4171:4171 -p 4000:4000 -p 4001:4001 -p 4002:4002 --name=%p --env-file=/etc/opsee/bastion-env.sh --cap-add=NET_ADMIN --device=/dev/net/tun --volume=/gozer/state --link bastion-etcd:etcd --link nsqd:nsqd --dns=${DNS_SERVER} quay.io/opsee/gozer:latest client\r\nExecStart=/usr/bin/docker logs -f %p\r\nExecStop=/usr/bin/docker stop -t 5 %p\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n```\r\n\r\n> On the pull, (ExecStartPre=/usr/bin/docker pull quay.io/opsee/gozer:latest), docker panics.  \r\n\r\n\r\n```\r\nDec 16 21:44:20 ip-172-31-52-56.ec2.internal dockerd[856]: time=\"2015-12-16T21:44:20.207364876Z\" level=error msg=\"Error from V2 registry: Server error: 404 trying to pull opsee/gozer blob - sha256:2574e0aab087b35c447ada7681c5d545a512dbdd8127df4d9f44dfed7f67dfd2\"\r\nDec 16 21:44:20 ip-172-31-52-56.ec2.internal systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT\r\nDec 16 21:44:20 ip-172-31-52-56.ec2.internal systemd[1]: docker.service: Unit entered failed state.\r\nDec 16 21:44:20 ip-172-31-52-56.ec2.internal systemd[1]: docker.service: Failed with result 'exit-code'.\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: panic: runtime error: invalid memory address or nil pointer dereference\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: [signal 0xb code=0x1 addr=0x20 pc=0x62f6fa]\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 554 [running]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: bufio.(*Writer).flush(0xc208275bc0, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/bufio/bufio.go:530 +0xda\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: bufio.(*Writer).Flush(0xc208275bc0, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/bufio/bufio.go:519 +0x3a\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: net/http.(*response).Flush(0xc2084b1540)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/net/http/server.go:1047 +0x4c\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc20821a5a0, 0xc208475ce0, 0xbb, 0x148, 0xbb, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:21 +0x145\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/pkg/progressreader.(*Config).Read(0xc208636700, 0xc2086c0000, 0x8000, 0x8000, 0x400, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/pkg/progressreader/progressreader.go:37 +0x2e5\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: io.Copy(0x7f038c9692a8, 0xc208039000, 0x7f038c97b798, 0xc208636700, 0xf787a, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/io/io.go:362 +0x1f6\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/graph.func·008(0xc2081d24f0, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/graph/pull.go:602 +0xd42\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/graph.func·009(0xc2081d24f0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/graph/pull.go:626 +0x2f\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: created by github.com/docker/docker/graph.(*TagStore).pullV2Tag\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/graph/pull.go:627 +0x2671\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 1 [chan receive]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: main.mainDaemon()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/docker/daemon.go:179 +0x10a8\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: main.main()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/docker/docker.go:93 +0x6b4\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 5 [syscall]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: os/signal.loop()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/os/signal/signal_unix.go:21 +0x1f\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: created by os/signal.init·1\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/os/signal/signal_unix.go:27 +0x35\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 17 [syscall, locked to thread]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: runtime.goexit()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 46 [chan receive]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: database/sql.(*DB).connectionOpener(0xc2081917c0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/database/sql/sql.go:589 +0x4c\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: created by database/sql.Open\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/database/sql/sql.go:452 +0x31c\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 9 [chan receive]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/api/server.(*Server).ServeApi(0xc208036f00, 0xc2080ad840, 0x1, 0x1, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/api/server/server.go:113 +0x74f\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: main.func·003()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/docker/daemon.go:129 +0x5f\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: created by main.mainDaemon\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/docker/daemon.go:135 +0x6b6\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 10 [chan receive]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/docker/docker/daemon.func·021()\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/daemon/debugtrap.go:17 +0x5c\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: created by github.com/docker/docker/daemon.setupSigusr1Trap\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/.gopath/src/github.com/docker/docker/daemon/debugtrap.go:20 +0x18e\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: goroutine 47 [runnable]:\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: net.(*pollDesc).Wait(0xc2081a3790, 0x72, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: net.(*pollDesc).WaitRead(0xc2081a3790, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: net.(*netFD).readMsg(0xc2081a3730, 0xc20861a010, 0x10, 0x10, 0xc20840d220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: net.(*UnixConn).ReadMsgUnix(0xc208039318, 0xc20861a010, 0x10, 0x10, 0xc20840d220, 0x1000, 0x1000, 0x36, 0xc208373d74, 0x4, ...)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/godbus/dbus.(*oobReader).Read(0xc20840d200, 0xc20861a010, 0x10, 0x10, 0xc20840d200, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/vendor/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: io.ReadAtLeast(0x7f038c971b98, 0xc20840d200, 0xc20861a010, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/io/io.go:298 +0xf1\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: io.ReadFull(0x7f038c971b98, 0xc20840d200, 0xc20861a010, 0x10, 0x10, 0x36, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /usr/lib/go/src/io/io.go:316 +0x6d\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc2080a6d80, 0xc20821af90, 0x0, 0x0)\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.7.1-r4/work/docker-1.7.1/vendor/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf\r\nDec 16 21:44:21 ip-172-31-52-56.ec2.internal dockerd[856]: github.com/godbus/dbus.(*Conn).inWorker(0xc2080f3680)\r\n```\r\n\r\n\r\n\r\n\r\ndocker version: \r\n```\r\nClient version: 1.7.1\r\nClient API version: 1.19\r\nGo version (client): go1.4.2\r\nGit commit (client): df2f73d-dirty\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.7.1\r\nServer API version: 1.19\r\nGo version (server): go1.4.2\r\nGit commit (server): df2f73d-dirty\r\nOS/Arch (server): linux/amd64\r\n```\r\n\r\nCoreOs Version\r\n```\r\n/etc/coreos $ cat update.conf\r\nGROUP=stable\r\n```"}}