{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarm/issues/342","labels_url":"https://api.github.com/repos/docker/swarm/issues/342/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarm/issues/342/comments","events_url":"https://api.github.com/repos/docker/swarm/issues/342/events","html_url":"https://github.com/docker/swarm/issues/342","id":56148506,"number":342,"title":"Leaking file descriptors","user":{"login":"jhamrick","id":83444,"avatar_url":"https://avatars.githubusercontent.com/u/83444?v=3","gravatar_id":"","url":"https://api.github.com/users/jhamrick","html_url":"https://github.com/jhamrick","followers_url":"https://api.github.com/users/jhamrick/followers","following_url":"https://api.github.com/users/jhamrick/following{/other_user}","gists_url":"https://api.github.com/users/jhamrick/gists{/gist_id}","starred_url":"https://api.github.com/users/jhamrick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhamrick/subscriptions","organizations_url":"https://api.github.com/users/jhamrick/orgs","repos_url":"https://api.github.com/users/jhamrick/repos","events_url":"https://api.github.com/users/jhamrick/events{/privacy}","received_events_url":"https://api.github.com/users/jhamrick/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-01T02:10:57Z","updated_at":"2015-02-06T03:15:55Z","closed_at":"2015-02-03T01:16:17Z","body":"It seems as though swarm is leaking file descriptors. Earlier today, I started seeing a bunch of errors like this in the swarm log:\r\n\r\n```\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 5ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 10ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 20ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 40ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 80ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 160ms\r\n2015/02/01 01:20:33 http: Accept error: accept tcp 127.0.0.1:2735: too many open files; retrying in 320ms\r\n```\r\n\r\nI only had about 80 containers running, but checking `lsof` revealed that the swarm process had 1035 file descriptors open (which is above the current limit of 1024). It would appear that whenever I stop or start a container, swarm opens a new fd, and then never closes it.\r\n\r\nEdit: Also, to clarify, restarting swarm fixed the problem. However, I imagine it will appear again next time swarm reaches >1024 file descriptors."},"comment":{"url":"https://api.github.com/repos/docker/swarm/issues/comments/73176646","html_url":"https://github.com/docker/swarm/issues/342#issuecomment-73176646","issue_url":"https://api.github.com/repos/docker/swarm/issues/342","id":73176646,"user":{"login":"jhamrick","id":83444,"avatar_url":"https://avatars.githubusercontent.com/u/83444?v=3","gravatar_id":"","url":"https://api.github.com/users/jhamrick","html_url":"https://github.com/jhamrick","followers_url":"https://api.github.com/users/jhamrick/followers","following_url":"https://api.github.com/users/jhamrick/following{/other_user}","gists_url":"https://api.github.com/users/jhamrick/gists{/gist_id}","starred_url":"https://api.github.com/users/jhamrick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhamrick/subscriptions","organizations_url":"https://api.github.com/users/jhamrick/orgs","repos_url":"https://api.github.com/users/jhamrick/repos","events_url":"https://api.github.com/users/jhamrick/events{/privacy}","received_events_url":"https://api.github.com/users/jhamrick/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T03:15:55Z","updated_at":"2015-02-06T03:15:55Z","body":"I'm not sure if this is fixed, unfortunately, as descriptors still seem to be counting up when containers are run and stopped:\r\n\r\n```\r\nroot@compmodels-hub-test:~# lsof -a -p $(pidof swarm) | wc -l\r\n14\r\nroot@compmodels-hub-test:~# docker --tls -H 127.0.0.1:2376 run hello-world\r\nHello from Docker.\r\nThis message shows that your installation appears to be working correctly.\r\n...\r\nroot@compmodels-hub-test:~# lsof -a -p $(pidof swarm) | wc -l\r\n19\r\nroot@compmodels-hub-test:~# docker --tls -H 127.0.0.1:2376 run hello-world\r\nHello from Docker.\r\nThis message shows that your installation appears to be working correctly.\r\n...\r\nroot@compmodels-hub-test:~# lsof -a -p $(pidof swarm) | wc -l\r\n22\r\n```\r\n\r\nThe full output of `lsof` shows that the connections are still in the `ESTABLISHED` state:\r\n\r\n```\r\nroot@compmodels-hub-test:~# lsof -a -p $(pidof swarm)\r\nCOMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME\r\nswarm   12192 root  cwd    DIR   0,39     4096     12 /go\r\nswarm   12192 root  rtd    DIR   0,39     4096      2 /\r\nswarm   12192 root  txt    REG   0,39  8971832     35 /srv/gocode/bin/swarm\r\nswarm   12192 root  mem    REG   0,39              45 /lib/x86_64-linux-gnu/libc-2.19.so (path dev=202,1, inode=918028)\r\nswarm   12192 root  mem    REG   0,39              43 /lib/x86_64-linux-gnu/libpthread-2.19.so (path dev=202,1, inode=918031)\r\nswarm   12192 root  mem    REG   0,39              40 /lib/x86_64-linux-gnu/ld-2.19.so (path dev=202,1, inode=917843)\r\nswarm   12192 root    0r   CHR    1,3      0t0 347992 /dev/null\r\nswarm   12192 root    1w  FIFO    0,8      0t0 348383 pipe\r\nswarm   12192 root    2w  FIFO    0,8      0t0 348384 pipe\r\nswarm   12192 root    3u  IPv4 348408      0t0    TCP ip6-localhost:2376 (LISTEN)\r\nswarm   12192 root    4u  0000    0,9        0   7623 anon_inode\r\nswarm   12192 root    5u  IPv4 348410      0t0    TCP compmodels-hub-test:54497->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root    6r   CHR    1,9      0t0 347996 /dev/urandom\r\nswarm   12192 root    9u  IPv4 348104      0t0    TCP compmodels-hub-test:54508->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   10u  IPv4 348100      0t0    TCP compmodels-hub-test:54502->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   11u  IPv4 348101      0t0    TCP compmodels-hub-test:54503->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   12u  IPv4 348442      0t0    TCP compmodels-hub-test:54510->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   13u  IPv4 348443      0t0    TCP compmodels-hub-test:54511->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   14u  IPv4 348142      0t0    TCP compmodels-hub-test:54517->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   15u  IPv4 348138      0t0    TCP compmodels-hub-test:54515->104.130.13.116:2376 (ESTABLISHED)\r\nswarm   12192 root   16u  IPv4 348477      0t0    TCP compmodels-hub-test:54518->104.130.13.116:2376 (ESTABLISHED)\r\n```"}}