{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/15204","labels_url":"https://api.github.com/repos/docker/docker/issues/15204/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/15204/comments","events_url":"https://api.github.com/repos/docker/docker/issues/15204/events","html_url":"https://github.com/docker/docker/issues/15204","id":98396303,"number":15204,"title":"/proc hung - unable to kill container and daemon","user":{"login":"hermansc","id":116978,"avatar_url":"https://avatars.githubusercontent.com/u/116978?v=3","gravatar_id":"","url":"https://api.github.com/users/hermansc","html_url":"https://github.com/hermansc","followers_url":"https://api.github.com/users/hermansc/followers","following_url":"https://api.github.com/users/hermansc/following{/other_user}","gists_url":"https://api.github.com/users/hermansc/gists{/gist_id}","starred_url":"https://api.github.com/users/hermansc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hermansc/subscriptions","organizations_url":"https://api.github.com/users/hermansc/orgs","repos_url":"https://api.github.com/users/hermansc/repos","events_url":"https://api.github.com/users/hermansc/events{/privacy}","received_events_url":"https://api.github.com/users/hermansc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/status/more-info-needed","name":"status/more-info-needed","color":"C6A49A"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-07-31T13:58:15Z","updated_at":"2015-12-14T16:47:25Z","closed_at":null,"body":"Hi.\r\n\r\nI've been running a container which now has put my system in a funny state. I first noticed it by not being able to do `ps -aux`, because it just hangs forever. I did `strace` and I got the following output:\r\n\r\n```\r\nstat(\"/proc/20262\", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0\r\nopen(\"/proc/20262/stat\", O_RDONLY)      = 6\r\nread(6, \"20262 (java) D 20258 20258 20258\"..., 1024) = 226\r\nclose(6)                                = 0\r\nopen(\"/proc/20262/status\", O_RDONLY)    = 6\r\nread(6, \"Name:\\tjava\\nState:\\tD (disk sleep)\"..., 1024) = 801\r\nclose(6)                                = 0\r\nopen(\"/proc/20262/cmdline\", O_RDONLY)   = 6\r\nread(6,\r\n```\r\n\r\nAnd there it hangs. So trying to read PID 20262's cmdline blocks the call. Reading [this blog post](https://rachelbythebay.com/w/2014/10/27/ps/) I think I know why:\r\n\r\n> Some process managers have decided they would rather be the ones who keep tabs on memory size and do the killing themselves. To do this, they run a task in a container, set a limit, and disable the kernel's oomkiller behavior in that container. Then they wait for notifications of it getting too big and kill it themselves.\r\n> This is all well and good, but have you ever seen what happens when the process manager fails? Life gets ... interesting.\r\n\r\nSo I'm suspecting the docker daemon has gotten itself in a pickle and now this is effecting the rest of my system. I'd prefer not to restart the docker daemon, as that would interrupt business critical services running there already -- but if forced to I'd at least love to know how I got in this state so I don't do the same again.\r\n\r\nHere's the output from running `docker inspect <container>`:\r\n\r\n```json\r\n{\r\n    \"AppArmorProfile\": \"\",\r\n    \"Args\": [\r\n        \"-c\",\r\n        \" /usr/lib/spark/bin/spark-class org.apache.spark.executor.MesosExecutorBackend\"\r\n    ],\r\n    \"Config\": {\r\n        \"AttachStderr\": false,\r\n        \"AttachStdin\": false,\r\n        \"AttachStdout\": false,\r\n        \"Cmd\": [\r\n            \"-c\",\r\n            \" /usr/lib/spark/bin/spark-class org.apache.spark.executor.MesosExecutorBackend\"\r\n        ],\r\n        \"CpuShares\": 2048,\r\n        \"Cpuset\": \"\",\r\n        \"Domainname\": \"\",\r\n        \"Entrypoint\": [\r\n            \"/bin/sh\"\r\n        ],\r\n        \"Env\": [\r\n            \"LIBPROCESS_PORT=0\",\r\n            \"MESOS_CHECKPOINT=0\",\r\n            \"MESOS_DIRECTORY=/local/mesos/work/slaves/20150516-115645-1209730570-5050-2077-S1/frameworks/20150518-110443-1243285002-5050-3534-19520/executors/20150516-115645-1209730570-5050-2077-S1/runs/25bb51b6-752b-4168-a516-ceca9ae03aae\",\r\n            \"MESOS_EXECUTOR_ID=20150516-115645-1209730570-5050-2077-S1\",\r\n            \"MESOS_FRAMEWORK_ID=20150518-110443-1243285002-5050-3534-19520\",\r\n            \"MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.22.1.so\",\r\n            \"MESOS_NATIVE_LIBRARY=/usr/lib/libmesos-0.22.1.so\",\r\n            \"MESOS_SLAVE_ID=20150516-115645-1209730570-5050-2077-S1\",\r\n            \"MESOS_SLAVE_PID=slave(1)@xxxxxxx:5051\",\r\n            \"SPARK_EXECUTOR_MEMORY=4096m\",\r\n            \"SPARK_EXECUTOR_OPTS=\",\r\n            \"SPARK_USER=root\",\r\n            \"SPARK_EXECUTOR_OPTS=\",\r\n            \"SPARK_USER=root\",\r\n            \"SPARK_EXECUTOR_MEMORY=4096m\",\r\n            \"MESOS_SANDBOX=/mnt/mesos/sandbox\",\r\n            \"PATH=/usr/lib/spark/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\r\n            \"HOME=/root\",\r\n            \"JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64\"\r\n        ],\r\n        \"ExposedPorts\": null,\r\n        \"Hostname\": \"hostname\",\r\n        \"Image\": \"apache/spark:1.4.0\",\r\n        \"MacAddress\": \"\",\r\n        \"Memory\": 4723834880,\r\n        \"MemorySwap\": -1,\r\n        \"NetworkDisabled\": false,\r\n        \"OnBuild\": null,\r\n        \"OpenStdin\": false,\r\n        \"PortSpecs\": null,\r\n        \"StdinOnce\": false,\r\n        \"Tty\": false,\r\n        \"User\": \"\",\r\n        \"Volumes\": null,\r\n        \"WorkingDir\": \"\"\r\n    },\r\n    \"Created\": \"2015-07-31T08:52:37.635002186Z\",\r\n    \"Driver\": \"aufs\",\r\n    \"ExecDriver\": \"native-0.2\",\r\n    \"ExecIDs\": [\r\n        \"613c2bde550aa451215657b9310ba1a5a9997f2eaf8f5797fa1e5f652e16ee18\"\r\n    ],\r\n    \"HostConfig\": {\r\n        \"Binds\": [\r\n            \"/usr/local/lib/libmesos.so:/usr/local/lib/libmesos.so:ro\",\r\n            \"/local/mesos/work/slaves/20150516-115645-1209730570-5050-2077-S1/frameworks/20150518-110443-1243285002-5050-3534-19520/executors/20150516-115645-1209730570-5050-2077-S1/runs/25bb51b6-752b-4168-a516-ceca9ae03aae:/mnt/mesos/sandbox\"\r\n        ],\r\n        \"CapAdd\": null,\r\n        \"CapDrop\": null,\r\n        \"ContainerIDFile\": \"\",\r\n        \"Devices\": [],\r\n        \"Dns\": null,\r\n        \"DnsSearch\": null,\r\n        \"ExtraHosts\": null,\r\n        \"IpcMode\": \"\",\r\n        \"Links\": null,\r\n        \"LxcConf\": [],\r\n        \"NetworkMode\": \"host\",\r\n        \"PidMode\": \"\",\r\n        \"PortBindings\": {},\r\n        \"Privileged\": false,\r\n        \"PublishAllPorts\": false,\r\n        \"ReadonlyRootfs\": false,\r\n        \"RestartPolicy\": {\r\n            \"MaximumRetryCount\": 0,\r\n            \"Name\": \"\"\r\n        },\r\n        \"SecurityOpt\": null,\r\n        \"VolumesFrom\": null\r\n    },\r\n    \"HostnamePath\": \"/local/docker/containers/158a48817335ef1c49b11cf02237cbda34997e8116055e6b63543e62aa1a1308/hostname\",\r\n    \"HostsPath\": \"/local/docker/containers/158a48817335ef1c49b11cf02237cbda34997e8116055e6b63543e62aa1a1308/hosts\",\r\n    \"Id\": \"158a48817335ef1c49b11cf02237cbda34997e8116055e6b63543e62aa1a1308\",\r\n    \"Image\": \"e246fb601d8f6e505b3264bc3fa3f1641ffca8dd80ae985588d4d451ae188a0f\",\r\n    \"MountLabel\": \"\",\r\n    \"Name\": \"/mesos-25bb51b6-752b-4168-a516-ceca9ae03aae\",\r\n    \"NetworkSettings\": {\r\n        \"Bridge\": \"\",\r\n        \"Gateway\": \"\",\r\n        \"GlobalIPv6Address\": \"\",\r\n        \"GlobalIPv6PrefixLen\": 0,\r\n        \"IPAddress\": \"\",\r\n        \"IPPrefixLen\": 0,\r\n        \"IPv6Gateway\": \"\",\r\n        \"LinkLocalIPv6Address\": \"\",\r\n        \"LinkLocalIPv6PrefixLen\": 0,\r\n        \"MacAddress\": \"\",\r\n        \"PortMapping\": null,\r\n        \"Ports\": null\r\n    },\r\n    \"Path\": \"/bin/sh\",\r\n    \"ProcessLabel\": \"\",\r\n    \"ResolvConfPath\": \"/local/docker/containers/158a48817335ef1c49b11cf02237cbda34997e8116055e6b63543e62aa1a1308/resolv.conf\",\r\n    \"RestartCount\": 0,\r\n    \"State\": {\r\n        \"Error\": \"\",\r\n        \"ExitCode\": 0,\r\n        \"FinishedAt\": \"0001-01-01T00:00:00Z\",\r\n        \"OOMKilled\": false,\r\n        \"Paused\": false,\r\n        \"Pid\": 20258,\r\n        \"Restarting\": false,\r\n        \"Running\": true,\r\n        \"StartedAt\": \"2015-07-31T08:52:38.712678631Z\"\r\n    },\r\n    \"Volumes\": {\r\n        \"/mnt/mesos/sandbox\": \"/local/mesos/work/slaves/20150516-115645-1209730570-5050-2077-S1/frameworks/20150518-110443-1243285002-5050-3534-19520/executors/20150516-115645-1209730570-5050-2077-S1/runs/25bb51b6-752b-4168-a516-ceca9ae03aae\",\r\n        \"/usr/local/lib/libmesos.so\": \"/usr/lib/libmesos-0.22.1.so\"\r\n    },\r\n    \"VolumesRW\": {\r\n        \"/mnt/mesos/sandbox\": true,\r\n        \"/usr/local/lib/libmesos.so\": false\r\n    }\r\n}\r\n```\r\n\r\nAnd the last output from `docker logs --tail=1 <container>`:\r\n\r\n```\r\nI0731 09:20:18.223043    70 exec.cpp:379] Executor asked to shutdown\r\n```\r\n\r\nWhich is a somewhat mesos specific message, but it should indicate that the docker container should be in a state where it's about to shutdown. And just to confirm, since you can see that the PID of the docker container itself is 20258 - the hanging PID (20262) is indeed the child of this process:\r\n\r\n```\r\n$ pgrep -P 20258\r\n20262\r\n```\r\n\r\nHere's the output from `docker -D info`:\r\n\r\n```\r\n$ sudo docker -D info\r\nContainers: 38\r\nImages: 1057\r\nStorage Driver: aufs\r\n Root Dir: /local/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 1293\r\nExecution Driver: native-0.2\r\nKernel Version: 3.13.0-29-generic\r\nOperating System: Ubuntu 14.04 LTS\r\nCPUs: 16\r\nTotal Memory: 18.9 GiB\r\nName: hostname\r\nID: QKNF:WVIG:DEPM:VKSW:57WQ:OIVP:6EHK:WNB6:RQME:STMT:YFLX:N5IY\r\nDebug mode (server): false\r\nDebug mode (client): true\r\nFds: 100172\r\nGoroutines: 600660\r\nEventsListeners: 1\r\nInit Path: /usr/bin/docker\r\nDocker Root Dir: /local/docker\r\nWARNING: No swap limit support\r\n\r\n$ docker version\r\nClient version: 1.5.0\r\nClient API version: 1.17\r\nGo version (client): go1.4.1\r\nGit commit (client): a8a31ef\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.5.0\r\nServer API version: 1.17\r\nGo version (server): go1.4.1\r\nGit commit (server): a8a31ef\r\n\r\n$ uname -a\r\nLinux hostname 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\nI'm sorry for not knowing how to reproduce this state, nor am I sure if the similar issue already exists in the bug tracker. Issue #9605 and #13885 seems related, but doesn't quite match my issue."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/164491020","html_url":"https://github.com/docker/docker/issues/15204#issuecomment-164491020","issue_url":"https://api.github.com/repos/docker/docker/issues/15204","id":164491020,"user":{"login":"talos","id":415831,"avatar_url":"https://avatars.githubusercontent.com/u/415831?v=3","gravatar_id":"","url":"https://api.github.com/users/talos","html_url":"https://github.com/talos","followers_url":"https://api.github.com/users/talos/followers","following_url":"https://api.github.com/users/talos/following{/other_user}","gists_url":"https://api.github.com/users/talos/gists{/gist_id}","starred_url":"https://api.github.com/users/talos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talos/subscriptions","organizations_url":"https://api.github.com/users/talos/orgs","repos_url":"https://api.github.com/users/talos/repos","events_url":"https://api.github.com/users/talos/events{/privacy}","received_events_url":"https://api.github.com/users/talos/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T16:47:25Z","updated_at":"2015-12-14T16:47:25Z","body":"I experienced the same issue with docker 1.9.1.  `ps aux` hangs.  `docker ps` and `docker info` also hung.  I was unable to restart the docker service, although after trying it seemed it died on its own -- although that didn't fix `ps aux`.\r\n\r\nUnfortunately, I can't get more specific debug info right now as the machine became unresponsive after an attempted restart -- it seems to have shut off all services but gotten stuck on Docker."}}