{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/12540","labels_url":"https://api.github.com/repos/saltstack/salt/issues/12540/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/12540/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/12540/events","html_url":"https://github.com/saltstack/salt/issues/12540","id":32806323,"number":12540,"title":"Application-level Keepalive is mandatory for healthy connections","user":{"login":"sivann","id":1550808,"avatar_url":"https://avatars.githubusercontent.com/u/1550808?v=3","gravatar_id":"","url":"https://api.github.com/users/sivann","html_url":"https://github.com/sivann","followers_url":"https://api.github.com/users/sivann/followers","following_url":"https://api.github.com/users/sivann/following{/other_user}","gists_url":"https://api.github.com/users/sivann/gists{/gist_id}","starred_url":"https://api.github.com/users/sivann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sivann/subscriptions","organizations_url":"https://api.github.com/users/sivann/orgs","repos_url":"https://api.github.com/users/sivann/repos","events_url":"https://api.github.com/users/sivann/events{/privacy}","received_events_url":"https://api.github.com/users/sivann/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Feature","name":"Feature","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2522,"closed_issues":2549,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-09-04T21:47:49Z","due_on":null,"closed_at":null},"comments":55,"created_at":"2014-05-05T12:49:27Z","updated_at":"2015-09-05T07:02:13Z","closed_at":null,"body":"Hello,\r\nThe problem:\r\n-------------------\r\nwe have now about 100 salt-minions which are installed in remote areas with 3G and satellite connections.\r\n\r\nWe loose connectivity with all of those minions in about 1-2 days after installation, with test.ping reporting \"minion did not return\". The state was each time that the minions saw an ESTABLISHED TCP connection, while on the salt-master there were no connection listed at all. (Yes that is correct).  Tighter keepalive settings were tried with no result. (OS is linux)\r\nEach time, restarting the salt-minion fixes the problem immediately.\r\n\r\nObviously the connections are transparently proxied someplace, (who knows what happens with those SAT networks) so the whole tcp-keepalive mechanism of 0mq fails.\r\n\r\nSuggestion:\r\n-------------------\r\nSalt should handle this on the application level, so as to determine connection health and reconnect if needed by e.g. sending a dummy ping data periodically every e.g. 10 minutes and checking for valid reply. The only workarround we can see is restarting salt-minion hourly which is really ugly.\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/137922553","html_url":"https://github.com/saltstack/salt/issues/12540#issuecomment-137922553","issue_url":"https://api.github.com/repos/saltstack/salt/issues/12540","id":137922553,"user":{"login":"bymodude","id":12155807,"avatar_url":"https://avatars.githubusercontent.com/u/12155807?v=3","gravatar_id":"","url":"https://api.github.com/users/bymodude","html_url":"https://github.com/bymodude","followers_url":"https://api.github.com/users/bymodude/followers","following_url":"https://api.github.com/users/bymodude/following{/other_user}","gists_url":"https://api.github.com/users/bymodude/gists{/gist_id}","starred_url":"https://api.github.com/users/bymodude/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bymodude/subscriptions","organizations_url":"https://api.github.com/users/bymodude/orgs","repos_url":"https://api.github.com/users/bymodude/repos","events_url":"https://api.github.com/users/bymodude/events{/privacy}","received_events_url":"https://api.github.com/users/bymodude/received_events","type":"User","site_admin":false},"created_at":"2015-09-05T07:02:13Z","updated_at":"2015-09-05T07:02:13Z","body":"just wanted to add regarding Azure for the benefit of others that use Salt with Minions or Master in Azure Cloud, the default tcp timeout is 4 minutes on Azure, so setting tcp keepalive below that is essential, or increasing the timeout on Azure side as documented here: http://azure.microsoft.com/en-us/blog/new-configurable-idle-timeout-for-azure-load-balancer/ "}}