{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/16281","labels_url":"https://api.github.com/repos/docker/docker/issues/16281/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/16281/comments","events_url":"https://api.github.com/repos/docker/docker/issues/16281/events","html_url":"https://github.com/docker/docker/issues/16281","id":106391137,"number":16281,"title":"docker rm performance drop since 1.8.2","user":{"login":"ceecko","id":8225242,"avatar_url":"https://avatars.githubusercontent.com/u/8225242?v=3","gravatar_id":"","url":"https://api.github.com/users/ceecko","html_url":"https://github.com/ceecko","followers_url":"https://api.github.com/users/ceecko/followers","following_url":"https://api.github.com/users/ceecko/following{/other_user}","gists_url":"https://api.github.com/users/ceecko/gists{/gist_id}","starred_url":"https://api.github.com/users/ceecko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceecko/subscriptions","organizations_url":"https://api.github.com/users/ceecko/orgs","repos_url":"https://api.github.com/users/ceecko/repos","events_url":"https://api.github.com/users/ceecko/events{/privacy}","received_events_url":"https://api.github.com/users/ceecko/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-14T17:30:39Z","updated_at":"2015-09-14T20:59:01Z","closed_at":null,"body":"\r\nHi,\r\n\r\nI noticed that after an upgrade from docker `1.7.1` to `1.8.2` the performance of `docker rm` command has dropped significantly. I managed to reproduce the behavior on two identical AWS machines (m4.large):\r\n\r\n```\r\n# cat /etc/centos-release\r\nCentOS Linux release 7.1.1503 (Core)\r\n\r\n# uname -a\r\nLinux ip-172-31-20-201.eu-west-1.compute.internal 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n# lsblk\r\nNAME                         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT\r\nxvda                         202:0    0    8G  0 disk\r\n└─xvda1                      202:1    0    8G  0 part /\r\nloop0                          7:0    0  100G  0 loop\r\n└─docker-202:1-25219536-pool 253:0    0  100G  0 dm\r\nloop1                          7:1    0    2G  0 loop\r\n└─docker-202:1-25219536-pool 253:0    0  100G  0 dm\r\n```\r\n\r\nOn `v1.7.1` it takes `~1.2s` to remove a container.\r\n\r\n```\r\n# docker version\r\nClient version: 1.7.1\r\nClient API version: 1.19\r\nPackage Version (client): docker-1.7.1-108.el7.centos.x86_64\r\nGo version (client): go1.4.2\r\nGit commit (client): 3043001/1.7.1\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.7.1\r\nServer API version: 1.19\r\nPackage Version (server): docker-1.7.1-108.el7.centos.x86_64\r\nGo version (server): go1.4.2\r\nGit commit (server): 3043001/1.7.1\r\nOS/Arch (server): linux/amd64\r\n\r\n# docker -D info\r\nContainers: 0\r\nImages: 2\r\nStorage Driver: devicemapper\r\n Pool Name: docker-202:1-25219536-pool\r\n Pool Blocksize: 65.54 kB\r\n Backing Filesystem: xfs\r\n Data file: /dev/loop0\r\n Metadata file: /dev/loop1\r\n Data Space Used: 459.8 MB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 6.934 GB\r\n Metadata Space Used: 860.2 kB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.147 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Data loop file: /var/lib/docker/devicemapper/devicemapper/data\r\n Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n Library Version: 1.02.93-RHEL7 (2015-01-28)\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.10.0-229.11.1.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nCPUs: 2\r\nTotal Memory: 7.389 GiB\r\nName: ip-172-31-21-188.eu-west-1.compute.internal\r\nID: JSWW:5A52:NGZU:FM4K:E3F4:EGL3:77AR:DCZM:W6FS:ZBMS:SBA3:ZXR6\r\n\r\n# DATE1=$(date +\"%s%3N\"); docker run --rm debian:jessie echo \"\"; echo \"Took $(($(date +\"%s%3N\") - $DATE1))ms\"\r\nUsage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.\r\n\r\nTook 1214ms\r\n```\r\n\r\nOn `v1.8.2` it takes `~6.6s` to remove a container.\r\n\r\n```\r\n# docker version\r\nClient:\r\n Version:      1.8.2\r\n API version:  1.20\r\n Go version:   go1.4.2\r\n Git commit:   0a8c2e3\r\n Built:        Thu Sep 10 19:08:45 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.8.2\r\n API version:  1.20\r\n Go version:   go1.4.2\r\n Git commit:   0a8c2e3\r\n Built:        Thu Sep 10 19:08:45 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\n# docker -D info\r\nContainers: 0\r\nImages: 2\r\nStorage Driver: devicemapper\r\n Pool Name: docker-202:1-58183-pool\r\n Pool Blocksize: 65.54 kB\r\n Backing Filesystem: xfs\r\n Data file: /dev/loop0\r\n Metadata file: /dev/loop1\r\n Data Space Used: 1.973 GB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 5.361 GB\r\n Metadata Space Used: 1.61 MB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.146 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Data loop file: /var/lib/docker/devicemapper/devicemapper/data\r\n Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n Library Version: 1.02.93-RHEL7 (2015-01-28)\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.10.0-229.11.1.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nCPUs: 2\r\nTotal Memory: 7.389 GiB\r\nName: ip-172-31-20-201.eu-west-1.compute.internal\r\nID: W3WU:5JBX:L6FH:NMNF:BGGY:VXOG:37FA:5N4Y:YAEZ:6LWB:6UGF:JU46\r\n\r\n# DATE1=$(date +\"%s%3N\"); docker run --rm debian:jessie echo \"\"; echo \"Took $(($(date +\"%s%3N\") - $DATE1))ms\"\r\n\r\nTook 6651ms\r\n```\r\n\r\nThe logs from `v1.8.2` don't show any errors:\r\n```\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.319018994Z\" level=info msg=\"POST /v1.20/containers/create\"\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.474291469Z\" level=info msg=\"POST /v1.20/containers/aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3/attach?stderr=1&stdout=1&stream=1\"\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.474892538Z\" level=info msg=\"POST /v1.20/containers/aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3/start\"\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: device veth52edf8b entered promiscuous mode\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: IPv6: ADDRCONF(NETDEV_UP): veth52edf8b: link is not ready\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered forwarding state\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered forwarding state\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered disabled state\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth52edf8b: link becomes ready\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered forwarding state\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered forwarding state\r\nSep 14 17:21:49 ip-172-31-20-201 systemd: Starting docker container aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3.\r\nSep 14 17:21:49 ip-172-31-20-201 systemd: Started docker container aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3.\r\nSep 14 17:21:49 ip-172-31-20-201 systemd: Stopping docker container aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3.\r\nSep 14 17:21:49 ip-172-31-20-201 systemd: Stopped docker container aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3.\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.707192689Z\" level=info msg=\"POST /v1.20/containers/aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3/wait\"\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered disabled state\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: device veth52edf8b left promiscuous mode\r\nSep 14 17:21:49 ip-172-31-20-201 kernel: docker0: port 1(veth52edf8b) entered disabled state\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.848508505Z\" level=info msg=\"GET /v1.20/containers/aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3/json\"\r\nSep 14 17:21:49 ip-172-31-20-201 docker: time=\"2015-09-14T17:21:49.849767196Z\" level=info msg=\"DELETE /v1.20/containers/aa12a680eea65b6d5c90a9dd448735e569435210238966152f378da9e3933cc3?v=1\"\r\n```\r\n\r\nIs this an expected behavior of `v1.8.2` due to some changes?"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/140204289","html_url":"https://github.com/docker/docker/issues/16281#issuecomment-140204289","issue_url":"https://api.github.com/repos/docker/docker/issues/16281","id":140204289,"user":{"login":"cpuguy83","id":799078,"avatar_url":"https://avatars.githubusercontent.com/u/799078?v=3","gravatar_id":"","url":"https://api.github.com/users/cpuguy83","html_url":"https://github.com/cpuguy83","followers_url":"https://api.github.com/users/cpuguy83/followers","following_url":"https://api.github.com/users/cpuguy83/following{/other_user}","gists_url":"https://api.github.com/users/cpuguy83/gists{/gist_id}","starred_url":"https://api.github.com/users/cpuguy83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpuguy83/subscriptions","organizations_url":"https://api.github.com/users/cpuguy83/orgs","repos_url":"https://api.github.com/users/cpuguy83/repos","events_url":"https://api.github.com/users/cpuguy83/events{/privacy}","received_events_url":"https://api.github.com/users/cpuguy83/received_events","type":"User","site_admin":false},"created_at":"2015-09-14T20:59:01Z","updated_at":"2015-09-14T20:59:01Z","body":"Maybe this is due to bumping up the allocated space from 10GB to 100GB?\r\nping @rhvgoyal "}}