{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/28640","labels_url":"https://api.github.com/repos/saltstack/salt/issues/28640/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/28640/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/28640/events","html_url":"https://github.com/saltstack/salt/issues/28640","id":115509211,"number":28640,"title":"Stale connections to master return port 4506","user":{"login":"dsumsky","id":2776381,"avatar_url":"https://avatars.githubusercontent.com/u/2776381?v=3","gravatar_id":"","url":"https://api.github.com/users/dsumsky","html_url":"https://github.com/dsumsky","followers_url":"https://api.github.com/users/dsumsky/followers","following_url":"https://api.github.com/users/dsumsky/following{/other_user}","gists_url":"https://api.github.com/users/dsumsky/gists{/gist_id}","starred_url":"https://api.github.com/users/dsumsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsumsky/subscriptions","organizations_url":"https://api.github.com/users/dsumsky/orgs","repos_url":"https://api.github.com/users/dsumsky/repos","events_url":"https://api.github.com/users/dsumsky/events{/privacy}","received_events_url":"https://api.github.com/users/dsumsky/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"c7def8"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Pending%20Discussion","name":"Pending Discussion","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2669,"closed_issues":2889,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-11-09T17:17:44Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-11-06T13:55:45Z","updated_at":"2015-11-09T17:34:38Z","closed_at":null,"body":"I noticed that on one of my masters  there are piling up stale TCP connections from minions (Linux, Windows) to ```return``` port 4506.  There is  more than 1 such connection per minion.  There shouldn't be any if there is no job running on it.\r\n\r\nWith ```netstat``` command I can see that all connections are in ```ESTABLISHED``` state. On the minion I can see one connection to ```publish``` port. That's expected. Then, there are also some to ```return``` port but not so many as on the master.  When I stop salt-minion service ```return``` connections are closed (```TIME_WAIT``` state in ```netstat```)  only on that minion. Nevertheless they stay in ```ESTABLISHED``` state on the master. \r\n\r\nSalt master version:\r\n```\r\n           Salt: 2015.5.3\r\n         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)\r\n         Jinja2: 2.2.1\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.1.13\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.0.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.10\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.5.0\r\n           RAET: Not Installed\r\n            ZMQ: 4.0.5\r\n           Mako: Not Installed\r\n        Tornado: Not Installed\r\n```\r\n\r\nSalt minion version :\r\n```\r\n   Salt: 2015.5.3\r\n         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)\r\n         Jinja2: 2.2.1\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.4.6\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.0.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.10\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.3.1\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.5\r\n           Mako: Not Installed\r\n        Tornado: Not Installed\r\n```\r\n\r\nSalt master (IP = ```10.201.0.173```)  ```netstat``` output:\r\n```\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:45558         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:33825         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:43344         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:41295         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:57614         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:53136         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4505           10.201.30.166:48033         ESTABLISHED 6003/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:58496         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:58209         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:45230         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:34669         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:37913         ESTABLISHED 14891/python2.6\r\n```\r\n\r\nSalt minion (IP = ```10.201.30.160```, CentOS6) ```netstat``` output:\r\n```\r\ntcp        0      0 10.201.30.166:33825         10.201.0.173:4506           ESTABLISHED 2682/python2.6\r\ntcp        0      0 10.201.30.166:48033         10.201.0.173:4505           ESTABLISHED 2682/python2.6\r\n```\r\n\r\nWhen ```salt-minion``` service is stopped ```publish``` connection is closed on both ends  but ```return``` ones stay on master in ```ESTABLISHED``` state:\r\n```\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:45558         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:43344         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:41295         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:57614         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:53136         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:58496         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:58209         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:45230         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:34669         ESTABLISHED 14891/python2.6\r\ntcp        0      0 10.201.0.173:4506           10.201.30.166:37913         ESTABLISHED 14891/python2.6\r\n```\r\n\r\nDoes anybody have any explanation how this may happen? What may prevent to close connections on master when the other end, minion, is down? Why master process is not cleaning such connections up?\r\n\r\nThe problem is that it is slowing down communication with masters. The only workaround I have is to restart masters time to time to clean stale connections up."},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/155134026","html_url":"https://github.com/saltstack/salt/issues/28640#issuecomment-155134026","issue_url":"https://api.github.com/repos/saltstack/salt/issues/28640","id":155134026,"user":{"login":"dsumsky","id":2776381,"avatar_url":"https://avatars.githubusercontent.com/u/2776381?v=3","gravatar_id":"","url":"https://api.github.com/users/dsumsky","html_url":"https://github.com/dsumsky","followers_url":"https://api.github.com/users/dsumsky/followers","following_url":"https://api.github.com/users/dsumsky/following{/other_user}","gists_url":"https://api.github.com/users/dsumsky/gists{/gist_id}","starred_url":"https://api.github.com/users/dsumsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsumsky/subscriptions","organizations_url":"https://api.github.com/users/dsumsky/orgs","repos_url":"https://api.github.com/users/dsumsky/repos","events_url":"https://api.github.com/users/dsumsky/events{/privacy}","received_events_url":"https://api.github.com/users/dsumsky/received_events","type":"User","site_admin":false},"created_at":"2015-11-09T17:34:38Z","updated_at":"2015-11-09T17:34:38Z","body":"@cachedout , could anybody provide me an issue link  where a similar problem is described?\r\nUpgrade what? Masters are not a problem, minions may be."}}