{"action":"closed","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/18515","labels_url":"https://api.github.com/repos/saltstack/salt/issues/18515/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/18515/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/18515/events","html_url":"https://github.com/saltstack/salt/issues/18515","id":50199827,"number":18515,"title":"Multiple Windows Minion 'Established TCP' connections causing master to become unresponsive","user":{"login":"ajonesspin","id":1344108,"avatar_url":"https://avatars.githubusercontent.com/u/1344108?v=3","gravatar_id":"","url":"https://api.github.com/users/ajonesspin","html_url":"https://github.com/ajonesspin","followers_url":"https://api.github.com/users/ajonesspin/followers","following_url":"https://api.github.com/users/ajonesspin/following{/other_user}","gists_url":"https://api.github.com/users/ajonesspin/gists{/gist_id}","starred_url":"https://api.github.com/users/ajonesspin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajonesspin/subscriptions","organizations_url":"https://api.github.com/users/ajonesspin/orgs","repos_url":"https://api.github.com/users/ajonesspin/repos","events_url":"https://api.github.com/users/ajonesspin/events{/privacy}","received_events_url":"https://api.github.com/users/ajonesspin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Critical","name":"Critical","color":"eb6420"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High+Severity","name":"High Severity","color":"eb6420"},{"url":"https://api.github.com/repos/saltstack/salt/labels/in+progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Windows","name":"Windows","color":"fef2c0"}],"state":"closed","locked":false,"assignee":{"login":"UtahDave","id":306240,"avatar_url":"https://avatars.githubusercontent.com/u/306240?v=3","gravatar_id":"","url":"https://api.github.com/users/UtahDave","html_url":"https://github.com/UtahDave","followers_url":"https://api.github.com/users/UtahDave/followers","following_url":"https://api.github.com/users/UtahDave/following{/other_user}","gists_url":"https://api.github.com/users/UtahDave/gists{/gist_id}","starred_url":"https://api.github.com/users/UtahDave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/UtahDave/subscriptions","organizations_url":"https://api.github.com/users/UtahDave/orgs","repos_url":"https://api.github.com/users/UtahDave/repos","events_url":"https://api.github.com/users/UtahDave/events{/privacy}","received_events_url":"https://api.github.com/users/UtahDave/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":1799,"closed_issues":1357,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-01-05T21:55:03Z","due_on":null,"closed_at":null},"comments":16,"created_at":"2014-11-26T16:33:24Z","updated_at":"2015-01-05T21:55:03Z","closed_at":"2015-01-05T21:55:03Z","body":"Hello\r\nI have an issue which I think is a bug, whereby the Windows minions continually establish new TCP connections on the salt-master server until the number of connections hits some limit and salt-master stops responding to any further requests.\r\n(This has started happening since update to 2014.7.0)\r\n\r\nIn my case this limit is approx. 4100 established TCP connections\r\n\r\nnetstat -a |grep ESTABLISHED |wc -l\r\n4117\r\n\r\nOnce this limit is hit (after approx. 2hrs from a salt-master restart), the CPU utilization reaches 100%\r\n\r\ntop\r\n  PID USER      PR  NI  VIRT     RES   SHR S  %CPU  %MEM    TIME+    COMMAND  \r\n51455 salt        20   0   1210m 125m  1576 S  99.2     6.7          31:19.18 salt-master\r\n\r\n\r\nand SYN_RECV connections are seen.\r\n\r\nnetstat -a |grep SYN_RECV |wc -l\r\n64\r\n\r\nI think the way I use Salt may highlight this issue quicker than in standard (primarlily state management) setups. Basically I have a few cron jobs which run BASH deployment scripts utilising Salt remote execution commands (stop/start services, transfer files etc.). These run every 10mins.\r\n\r\nThis issue never occured with version 2014.1.7. With that version, TCP connection on the master stayed at approx. 2000 continuously.\r\n\r\n\r\nSteps to Reproduce - \r\n\r\nWould need a script with salt remote execution commands against windows minions scheduled at regular intervals. Monitor TCP connections on salt master and observer increased count of established connections by the minions.\r\nWhen 4100 has been reached, the CPU utilization will become very high (although this limit may be specific to my environment). However, the TCP connections should not increase continuously.\r\n\r\n-------------\r\n\r\nEnv. details - \r\n\r\nCentos6.6 (x86-64)  - Virtual machine (VMWare, single CPU, 2GB RAM) - salt-master 2014.7.0\r\n\r\nWindows minions - Windows 2008 R2 (x64) / Windows 2012 - salt-minon 2014.7.0\r\napprox. 40 machines\r\n\r\nLinux minions  - Centos6.6 (x86-64) - salt-minon 2014.7.0\r\napprox. 15\r\n```\r\nsalt --versions-report\r\n           Salt: 2014.7.0\r\n         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)\r\n         Jinja2: 2.2.1\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.1.13\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.0.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.09\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.3.1\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.4\r\n           Mako: Not Installed\r\n```\r\n\r\nPlease let me know if you require more info or other investigation tasks.\r\n\r\nThanks\r\n\r\n\r\n"}}