{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/16748","labels_url":"https://api.github.com/repos/docker/docker/issues/16748/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/16748/comments","events_url":"https://api.github.com/repos/docker/docker/issues/16748/events","html_url":"https://github.com/docker/docker/issues/16748","id":109655771,"number":16748,"title":"1.9: Possible CPU issue from libnetwork","user":{"login":"ibuildthecloud","id":1754002,"avatar_url":"https://avatars.githubusercontent.com/u/1754002?v=3","gravatar_id":"","url":"https://api.github.com/users/ibuildthecloud","html_url":"https://github.com/ibuildthecloud","followers_url":"https://api.github.com/users/ibuildthecloud/followers","following_url":"https://api.github.com/users/ibuildthecloud/following{/other_user}","gists_url":"https://api.github.com/users/ibuildthecloud/gists{/gist_id}","starred_url":"https://api.github.com/users/ibuildthecloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibuildthecloud/subscriptions","organizations_url":"https://api.github.com/users/ibuildthecloud/orgs","repos_url":"https://api.github.com/users/ibuildthecloud/repos","events_url":"https://api.github.com/users/ibuildthecloud/events{/privacy}","received_events_url":"https://api.github.com/users/ibuildthecloud/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-04T01:24:48Z","updated_at":"2015-10-04T02:32:46Z","closed_at":"2015-10-04T01:29:33Z","body":"@mavenugo This may be the most useless bug report, but I have reason to believe the following gorouting is chewing up 100% CPU:\r\n\r\n```\r\ngoroutine 182 [select]:\r\nnet/http.(*Transport).getConn(0xc20829d4d0, 0xc208974410, 0x0, 0xc20865bc20, 0x4, 0xc20865bc27, 0xe, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/transport.go:525 +0x608\r\nnet/http.(*Transport).RoundTrip(0xc20829d4d0, 0xc208974410, 0xc20865bd10, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/transport.go:228 +0x4d4\r\nnet/http.send(0xc208974410, 0x7f2acd81c5f0, 0xc20829d4d0, 0xe4, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:219 +0x4fc\r\nnet/http.(*Client).send(0xc2082cb6b0, 0xc208974410, 0xe4, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:142 +0x15b\r\nnet/http.(*Client).doFollowingRedirects(0xc2082cb6b0, 0xc208974410, 0x1434ed0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:367 +0xb25\r\nnet/http.(*Client).Do(0xc2082cb6b0, 0xc208974410, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:174 +0xa4\r\ngithub.com/coreos/go-etcd/etcd.(*Client).SendRequest(0xc20829d440, 0xc20a44fd40, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:238 +0x121f\r\ngithub.com/coreos/go-etcd/etcd.(*Client).getCancelable(0xc20829d440, 0xc20a453860, 0x9e, 0xc20a44fc50, 0x0, 0x1a02fc0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:51 +0x41a\r\ngithub.com/coreos/go-etcd/etcd.(*Client).get(0xc20829d440, 0xc20a453860, 0x9e, 0xc20a44fc50, 0x4b1561, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:62 +0x61\r\ngithub.com/coreos/go-etcd/etcd.(*Client).RawGet(0xc20829d440, 0xc20a453860, 0x9e, 0x0, 0xc20a4537c0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/get.go:31 +0x26c\r\ngithub.com/coreos/go-etcd/etcd.(*Client).Get(0xc20829d440, 0xc20a453860, 0x9e, 0x0, 0x9d, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/get.go:11 +0x63\r\ngithub.com/docker/libkv/store/etcd.(*Etcd).Get(0xc208038788, 0xc20a453680, 0x9d, 0xc20a4535e0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:126 +0x86\r\ngithub.com/docker/libnetwork/datastore.(*datastore).GetObject(0xc2082ea410, 0xc20a453680, 0x9d, 0x7f2acd831c38, 0xc2089f8480, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/datastore/datastore.go:188 +0x70\r\ngithub.com/docker/libnetwork.func·042()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:244 +0x58b\r\ncreated by github.com/docker/libnetwork.(*network).watchEndpoints\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:253 +0x576\r\n```\r\n\r\nThe situation is such that I started with an etcd backend, but then killed the server.  So Docker should be getting a connection refused right now.\r\n\r\nTo help troubleshoot, I pushed a tag to by github account because this is docker master + your two UX PR's. https://github.com/ibuildthecloud/docker/blob/cpu-issue/vendor/src/github.com/docker/libnetwork/store.go#L244\r\n\r\nRight now Docker is using 100% CPU.  This is the full goroutine dump\r\n```\r\nINFO[84249] === BEGIN goroutine stack dump ===\r\ngoroutine 10 [running]:\r\ngithub.com/docker/docker/pkg/signal.DumpStacks()\r\n\t/go/src/github.com/docker/docker/pkg/signal/trap.go:67 +0xa5\r\ngithub.com/docker/docker/daemon.func·030()\r\n\t/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:18 +0x6d\r\ncreated by github.com/docker/docker/daemon.setupDumpStackTrap\r\n\t/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:20 +0x18e\r\n\r\ngoroutine 1 [chan receive, 386 minutes]:\r\nmain.(*DaemonCli).CmdDaemon(0xc2080f9ed0, 0xc20800a020, 0x1, 0x1, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/docker/daemon.go:286 +0x1867\r\nreflect.callMethod(0xc208068ae0, 0xc2082bfce8)\r\n\t/usr/local/go/src/reflect/value.go:605 +0x179\r\nreflect.methodValueCall(0xc20800a020, 0x1, 0x1, 0x1, 0xc208068ae0, 0x0, 0x0, 0xc208068ae0, 0x0, 0x4dbc0f, ...)\r\n\t/usr/local/go/src/reflect/asm_amd64.s:29 +0x36\r\ngithub.com/docker/docker/cli.(*Cli).Run(0xc208099170, 0xc20800a010, 0x2, 0x2, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/cli/cli.go:89 +0x38e\r\nmain.main()\r\n\t/go/src/github.com/docker/docker/docker/docker.go:65 +0x418\r\n\r\ngoroutine 5 [syscall]:\r\nos/signal.loop()\r\n\t/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f\r\ncreated by os/signal.init·1\r\n\t/usr/local/go/src/os/signal/signal_unix.go:27 +0x35\r\n\r\ngoroutine 17 [syscall, 386 minutes, locked to thread]:\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1\r\n\r\ngoroutine 12 [IO wait, 386 minutes]:\r\nnet.(*pollDesc).Wait(0xc2080da7d0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc2080da7d0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).readMsg(0xc2080da770, 0xc2082c8ef0, 0x10, 0x10, 0xc2082d6020, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)\r\n\t/usr/local/go/src/net/fd_unix.go:296 +0x54e\r\nnet.(*UnixConn).ReadMsgUnix(0xc208038478, 0xc2082c8ef0, 0x10, 0x10, 0xc2082d6020, 0x1000, 0x1000, 0x51, 0xc2082c8cb4, 0x4, ...)\r\n\t/usr/local/go/src/net/unixsock_posix.go:147 +0x167\r\ngithub.com/godbus/dbus.(*oobReader).Read(0xc2082d6000, 0xc2082c8ef0, 0x10, 0x10, 0xc2082d6000, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5\r\nio.ReadAtLeast(0x7f2acd826e58, 0xc2082d6000, 0xc2082c8ef0, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/io/io.go:298 +0xf1\r\nio.ReadFull(0x7f2acd826e58, 0xc2082d6000, 0xc2082c8ef0, 0x10, 0x10, 0x51, 0x0, 0x0)\r\n\t/usr/local/go/src/io/io.go:316 +0x6d\r\ngithub.com/godbus/dbus.(*unixTransport).ReadMessage(0xc2082a9210, 0xc2080f15c0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf\r\ngithub.com/godbus/dbus.(*Conn).inWorker(0xc2082a0c60)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/conn.go:248 +0x58\r\ncreated by github.com/godbus/dbus.(*Conn).Auth\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/auth.go:118 +0xe84\r\n\r\ngoroutine 13 [chan receive, 386 minutes]:\r\ngithub.com/godbus/dbus.(*Conn).outWorker(0xc2082a0c60)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/conn.go:370 +0x58\r\ncreated by github.com/godbus/dbus.(*Conn).Auth\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/auth.go:119 +0xea1\r\n\r\ngoroutine 14 [chan receive, 386 minutes]:\r\ngithub.com/docker/libnetwork/iptables.signalHandler()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/iptables/firewalld.go:92 +0x57\r\ncreated by github.com/docker/libnetwork/iptables.FirewalldInit\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/iptables/firewalld.go:48 +0x185\r\n\r\ngoroutine 26 [sleep, 6 minutes]:\r\ngithub.com/docker/libkv/store/etcd.func·001()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:78 +0x4a\r\ncreated by github.com/docker/libkv/store/etcd.New\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:80 +0x329\r\n\r\ngoroutine 44 [IO wait, 386 minutes]:\r\nnet.(*pollDesc).Wait(0xc20800e3e0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20800e3e0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc20800e380, 0xc208330000, 0x1000, 0x1000, 0x0, 0x7f2acd81c380, 0xc20838a6d8)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc208038c18, 0xc208330000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nnet/http.noteEOFReader.Read(0x7f2acd82b7c8, 0xc208038c18, 0xc2080c5bd8, 0xc208330000, 0x1000, 0x1000, 0xf69f40, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/transport.go:1270 +0x6e\r\nnet/http.(*noteEOFReader).Read(0xc20832c360, 0xc208330000, 0x1000, 0x1000, 0xc208010000, 0x0, 0x0)\r\n\t<autogenerated>:125 +0xd4\r\nbufio.(*Reader).fill(0xc2081b9860)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).Peek(0xc2081b9860, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:132 +0xf0\r\nnet/http.(*persistConn).readLoop(0xc2080c5b80)\r\n\t/usr/local/go/src/net/http/transport.go:842 +0xa4\r\ncreated by net/http.(*Transport).dialConn\r\n\t/usr/local/go/src/net/http/transport.go:660 +0xc9f\r\n\r\ngoroutine 82 [IO wait, 386 minutes]:\r\nnet.(*pollDesc).Wait(0xc2083be530, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc2083be530, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).accept(0xc2083be4d0, 0x0, 0x7f2acd81c380, 0xc2083c84b8)\r\n\t/usr/local/go/src/net/fd_unix.go:419 +0x40b\r\nnet.(*UnixListener).AcceptUnix(0xc2083c25a0, 0x10, 0x0, 0x0)\r\n\t/usr/local/go/src/net/unixsock_posix.go:282 +0x56\r\nnet.(*UnixListener).Accept(0xc2083c25a0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/unixsock_posix.go:293 +0x4c\r\ngithub.com/docker/libnetwork.(*controller).acceptClientConnections(0xc20808e900, 0xc2083be460, 0x63, 0x7f2acd82bdc8, 0xc2083c25a0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey.go:138 +0x87\r\ncreated by github.com/docker/libnetwork.(*controller).startExternalKeyListener\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey.go:132 +0x28f\r\n\r\ngoroutine 45 [select, 386 minutes]:\r\nnet/http.(*persistConn).writeLoop(0xc2080c5b80)\r\n\t/usr/local/go/src/net/http/transport.go:945 +0x41d\r\ncreated by net/http.(*Transport).dialConn\r\n\t/usr/local/go/src/net/http/transport.go:661 +0xcbc\r\n\r\ngoroutine 99 [chan receive, 386 minutes]:\r\ndatabase/sql.(*DB).connectionOpener(0xc2083e1360)\r\n\t/usr/local/go/src/database/sql/sql.go:589 +0x4c\r\ncreated by database/sql.Open\r\n\t/usr/local/go/src/database/sql/sql.go:452 +0x31c\r\n\r\ngoroutine 103 [chan receive, 1 minutes]:\r\ngithub.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc208065100)\r\n\t/go/src/github.com/docker/docker/daemon/exec.go:298 +0x8c\r\ncreated by github.com/docker/docker/daemon.NewDaemon\r\n\t/go/src/github.com/docker/docker/daemon/daemon.go:794 +0x2a27\r\n\r\ngoroutine 102 [chan receive]:\r\ngithub.com/docker/docker/daemon.(*statsCollector).run(0xc2083f9fb0)\r\n\t/go/src/github.com/docker/docker/daemon/stats_collector_unix.go:91 +0xb2\r\ncreated by github.com/docker/docker/daemon.newStatsCollector\r\n\t/go/src/github.com/docker/docker/daemon/stats_collector_unix.go:31 +0x116\r\n\r\ngoroutine 129 [select]:\r\ngithub.com/docker/libnetwork/osl.removeUnusedPaths()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:73 +0x48b\r\ncreated by github.com/docker/libnetwork/osl.createBasePath\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:58 +0xb1\r\n\r\ngoroutine 120 [IO wait, 72 minutes]:\r\nnet.(*pollDesc).Wait(0xc20847dfe0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20847dfe0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).accept(0xc20847df80, 0x0, 0x7f2acd81c380, 0xc208ce5638)\r\n\t/usr/local/go/src/net/fd_unix.go:419 +0x40b\r\nnet.(*UnixListener).AcceptUnix(0xc20847fa60, 0xc20889fb00, 0x0, 0x0)\r\n\t/usr/local/go/src/net/unixsock_posix.go:282 +0x56\r\nnet.(*UnixListener).Accept(0xc20847fa60, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/unixsock_posix.go:293 +0x4c\r\ngithub.com/docker/docker/pkg/listenbuffer.(*defaultListener).Accept(0xc20847fa80, 0x0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/pkg/listenbuffer/buffer.go:71 +0x67\r\nnet/http.(*Server).Serve(0xc2084c8000, 0x7f2acd830898, 0xc20847fa80, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1728 +0x92\r\ngithub.com/docker/docker/api/server.(*HTTPServer).Serve(0xc20843ca20, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/api/server/server.go:114 +0x4d\r\ngithub.com/docker/docker/api/server.func·006(0x7f2acd830b80, 0xc20843ca20)\r\n\t/go/src/github.com/docker/docker/api/server/server.go:86 +0x40\r\ncreated by github.com/docker/docker/api/server.(*Server).ServeAPI\r\n\t/go/src/github.com/docker/docker/api/server/server.go:90 +0x6ac\r\n\r\ngoroutine 118 [chan receive, 386 minutes]:\r\ngithub.com/docker/docker/pkg/signal.func·002()\r\n\t/go/src/github.com/docker/docker/pkg/signal/trap.go:29 +0x8f\r\ncreated by github.com/docker/docker/pkg/signal.Trap\r\n\t/go/src/github.com/docker/docker/pkg/signal/trap.go:55 +0x250\r\n\r\ngoroutine 117 [chan receive, 386 minutes]:\r\ngithub.com/docker/docker/api/server.(*Server).ServeAPI(0xc208467e00, 0xc20809a530, 0x1, 0x1, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/api/server/server.go:95 +0x74f\r\nmain.func·007()\r\n\t/go/src/github.com/docker/docker/docker/daemon.go:260 +0x5b\r\ncreated by main.(*DaemonCli).CmdDaemon\r\n\t/go/src/github.com/docker/docker/docker/daemon.go:266 +0x170f\r\n\r\ngoroutine 140 [sleep, 6 minutes]:\r\ngithub.com/docker/libkv/store/etcd.func·001()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:78 +0x4a\r\ncreated by github.com/docker/libkv/store/etcd.New\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:80 +0x329\r\n\r\ngoroutine 143 [runnable]:\r\ngithub.com/docker/libnetwork/bitseq.func·001()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/bitseq/store.go:99 +0x7a\r\ncreated by github.com/docker/libnetwork/bitseq.(*Handle).watchForChanges\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/bitseq/store.go:111 +0x281\r\n\r\ngoroutine 159 [runnable]:\r\ngithub.com/docker/libnetwork/bitseq.func·001()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/bitseq/store.go:99 +0x7a\r\ncreated by github.com/docker/libnetwork/bitseq.(*Handle).watchForChanges\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/bitseq/store.go:111 +0x281\r\n\r\ngoroutine 182 [select]:\r\nnet/http.(*Transport).getConn(0xc20829d4d0, 0xc208974410, 0x0, 0xc20865bc20, 0x4, 0xc20865bc27, 0xe, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/transport.go:525 +0x608\r\nnet/http.(*Transport).RoundTrip(0xc20829d4d0, 0xc208974410, 0xc20865bd10, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/transport.go:228 +0x4d4\r\nnet/http.send(0xc208974410, 0x7f2acd81c5f0, 0xc20829d4d0, 0xe4, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:219 +0x4fc\r\nnet/http.(*Client).send(0xc2082cb6b0, 0xc208974410, 0xe4, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:142 +0x15b\r\nnet/http.(*Client).doFollowingRedirects(0xc2082cb6b0, 0xc208974410, 0x1434ed0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:367 +0xb25\r\nnet/http.(*Client).Do(0xc2082cb6b0, 0xc208974410, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/client.go:174 +0xa4\r\ngithub.com/coreos/go-etcd/etcd.(*Client).SendRequest(0xc20829d440, 0xc20a44fd40, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:238 +0x121f\r\ngithub.com/coreos/go-etcd/etcd.(*Client).getCancelable(0xc20829d440, 0xc20a453860, 0x9e, 0xc20a44fc50, 0x0, 0x1a02fc0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:51 +0x41a\r\ngithub.com/coreos/go-etcd/etcd.(*Client).get(0xc20829d440, 0xc20a453860, 0x9e, 0xc20a44fc50, 0x4b1561, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/requests.go:62 +0x61\r\ngithub.com/coreos/go-etcd/etcd.(*Client).RawGet(0xc20829d440, 0xc20a453860, 0x9e, 0x0, 0xc20a4537c0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/get.go:31 +0x26c\r\ngithub.com/coreos/go-etcd/etcd.(*Client).Get(0xc20829d440, 0xc20a453860, 0x9e, 0x0, 0x9d, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/coreos/go-etcd/etcd/get.go:11 +0x63\r\ngithub.com/docker/libkv/store/etcd.(*Etcd).Get(0xc208038788, 0xc20a453680, 0x9d, 0xc20a4535e0, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libkv/store/etcd/etcd.go:126 +0x86\r\ngithub.com/docker/libnetwork/datastore.(*datastore).GetObject(0xc2082ea410, 0xc20a453680, 0x9d, 0x7f2acd831c38, 0xc2089f8480, 0x0, 0x0)\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/datastore/datastore.go:188 +0x70\r\ngithub.com/docker/libnetwork.func·042()\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:244 +0x58b\r\ncreated by github.com/docker/libnetwork.(*network).watchEndpoints\r\n\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:253 +0x576\r\n\r\ngoroutine 240 [select, 189 minutes]:\r\nnet.func·019()\r\n\t/usr/local/go/src/net/dnsclient_unix.go:241 +0x2f4\r\ncreated by net.loadConfig\r\n\t/usr/local/go/src/net/dnsclient_unix.go:269 +0x20c\r\n\r\ngoroutine 284727 [runnable]:\r\nnet/http.func·019()\r\n\t/usr/local/go/src/net/http/transport.go:519\r\ncreated by net/http.(*Transport).getConn\r\n\t/usr/local/go/src/net/http/transport.go:522 +0x335\r\n\r\n=== END goroutine stack dump === \r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/145311203","html_url":"https://github.com/docker/docker/issues/16748#issuecomment-145311203","issue_url":"https://api.github.com/repos/docker/docker/issues/16748","id":145311203,"user":{"login":"abronan","id":1655859,"avatar_url":"https://avatars.githubusercontent.com/u/1655859?v=3","gravatar_id":"","url":"https://api.github.com/users/abronan","html_url":"https://github.com/abronan","followers_url":"https://api.github.com/users/abronan/followers","following_url":"https://api.github.com/users/abronan/following{/other_user}","gists_url":"https://api.github.com/users/abronan/gists{/gist_id}","starred_url":"https://api.github.com/users/abronan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abronan/subscriptions","organizations_url":"https://api.github.com/users/abronan/orgs","repos_url":"https://api.github.com/users/abronan/repos","events_url":"https://api.github.com/users/abronan/events{/privacy}","received_events_url":"https://api.github.com/users/abronan/received_events","type":"User","site_admin":false},"created_at":"2015-10-04T02:32:46Z","updated_at":"2015-10-04T02:32:46Z","body":"@mavenugo I suspect that this is not a duplicate as I'm seeing the same issue with a fresh new daemon running `libnetwork` (ie. without restarting the store). The daemon takes up 100% CPU no matter what :confused: "}}