{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/26888","labels_url":"https://api.github.com/repos/saltstack/salt/issues/26888/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/26888/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/26888/events","html_url":"https://github.com/saltstack/salt/issues/26888","id":104739695,"number":26888,"title":"salt-minion error on unstable tcp connection","user":{"login":"bymodude","id":12155807,"avatar_url":"https://avatars.githubusercontent.com/u/12155807?v=3","gravatar_id":"","url":"https://api.github.com/users/bymodude","html_url":"https://github.com/bymodude","followers_url":"https://api.github.com/users/bymodude/followers","following_url":"https://api.github.com/users/bymodude/following{/other_user}","gists_url":"https://api.github.com/users/bymodude/gists{/gist_id}","starred_url":"https://api.github.com/users/bymodude/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bymodude/subscriptions","organizations_url":"https://api.github.com/users/bymodude/orgs","repos_url":"https://api.github.com/users/bymodude/repos","events_url":"https://api.github.com/users/bymodude/events{/privacy}","received_events_url":"https://api.github.com/users/bymodude/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-09-03T17:01:41Z","updated_at":"2015-09-05T17:27:09Z","closed_at":null,"body":"I have a minion connecting to a master via Internet, the routing between these 2 servers is bit unstable, e.g. there is some packet loss, however many other services are running fine, just salt is having bigger problems. \r\n\r\none server is in Azure Cloud and they dont allow ICMP, so here is a tcpping to give an idea of the packet loss encountered:\r\n```\r\nroot@minion1:~# ./tcpping salt-master.example.com 4506\r\nseq 0: tcp response from salt-master.example.com (1.1.1.1) [open]  310.518 ms\r\nseq 1: tcp response from salt-master.example.com (1.1.1.1) [open]  314.011 ms\r\nseq 2: tcp response from salt-master.example.com (1.1.1.1) [open]  310.209 ms\r\nseq 3: tcp response from salt-master.example.com (1.1.1.1) [open]  310.258 ms\r\nseq 5: tcp response from salt-master.example.com (1.1.1.1) [open]  310.698 ms\r\nseq 4: no response (timeout)\r\nseq 7: tcp response from salt-master.example.com (1.1.1.1) [open]  313.063 ms\r\nseq 8: tcp response from salt-master.example.com (1.1.1.1) [open]  309.968 ms\r\nseq 6: no response (timeout)\r\nseq 9: tcp response from salt-master.example.com (1.1.1.1) [open]  313.527 ms\r\nseq 10: tcp response from salt-master.example.com (1.1.1.1) [open]  311.129 ms\r\nseq 11: tcp response from salt-master.example.com (1.1.1.1) [open]  310.477 ms\r\nseq 12: tcp response from salt-master.example.com (1.1.1.1) [open]  313.292 ms\r\nseq 13: tcp response from salt-master.example.com (1.1.1.1) [open]  313.591 ms\r\nseq 15: tcp response from salt-master.example.com (1.1.1.1) [open]  314.750 ms\r\nseq 16: tcp response from salt-master.example.com (1.1.1.1) [open]  313.983 ms\r\nseq 14: no response (timeout)\r\nseq 17: tcp response from salt-master.example.com (1.1.1.1) [open]  310.181 ms\r\nseq 19: tcp response from salt-master.example.com (1.1.1.1) [open]  312.813 ms\r\n```\r\n\r\nhere is \"salt-minion -l debug\", trying \"salt minion1 test.ping\" from the master only works about 1 out of 10 times, just wondering if the errors below are normal in this situation of a not 100% stable connection and also if its normal to expect salt to work this badly over this kind of connection (pretty much unusable, whereas other services are working fine even though there is some packet loss)\r\n```\r\n[DEBUG   ] Ping master\r\n[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'minion1', 'tcp://1.1.1.1:4506')\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 3)\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 3)\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 3)\r\n[INFO    ] fire_master failed: Traceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 992, in _fire_master\r\n    result = channel.send(load, timeout=timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 314, in send\r\n    return self._crypted_transfer(load, tries, timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 302, in _crypted_transfer\r\n    return _do_transfer()\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 293, in _do_transfer\r\n    timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/payload.py\", line 273, in send\r\n    'SaltReqTimeoutError: after {0} seconds, ran {1} tries'.format(timeout * tried, tried)\r\nSaltReqTimeoutError: SaltReqTimeoutError: after 180 seconds, ran 3 tries\r\n\r\n[DEBUG   ] Ping master\r\n[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'minion1', 'tcp://1.1.1.1:4506')\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 3)\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 3)\r\n[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 3)\r\n[INFO    ] fire_master failed: Traceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 992, in _fire_master\r\n    result = channel.send(load, timeout=timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 314, in send\r\n    return self._crypted_transfer(load, tries, timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 302, in _crypted_transfer\r\n    return _do_transfer()\r\n  File \"/usr/lib/python2.7/dist-packages/salt/transport/__init__.py\", line 293, in _do_transfer\r\n    timeout)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/payload.py\", line 273, in send\r\n    'SaltReqTimeoutError: after {0} seconds, ran {1} tries'.format(timeout * tried, tried)\r\nSaltReqTimeoutError: SaltReqTimeoutError: after 180 seconds, ran 3 tries\r\n```\r\n\r\nI experience similar kind of problems with other minions, all are connecting via Internet to the master. You could argue that the connection in the above example is not good, but over WAN connecting multiple clouds and many servers its just expected to have a few of those not ideal connections, other services between those servers (mostly HTTP/HTTPS based etc) run fine, is it normal that ZMQ is this sensitive to packet loss, given that other services are able to recover without problems but with ZMQ I always seem to have minions loosing their connection to the master. \r\n\r\nI wanted to try RAET, but it has been broken for a few releases now already, I tried following the RAET bugs and the patches there but could never get it to work sadly.  "},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/137979360","html_url":"https://github.com/saltstack/salt/issues/26888#issuecomment-137979360","issue_url":"https://api.github.com/repos/saltstack/salt/issues/26888","id":137979360,"user":{"login":"steverweber","id":1800619,"avatar_url":"https://avatars.githubusercontent.com/u/1800619?v=3","gravatar_id":"","url":"https://api.github.com/users/steverweber","html_url":"https://github.com/steverweber","followers_url":"https://api.github.com/users/steverweber/followers","following_url":"https://api.github.com/users/steverweber/following{/other_user}","gists_url":"https://api.github.com/users/steverweber/gists{/gist_id}","starred_url":"https://api.github.com/users/steverweber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steverweber/subscriptions","organizations_url":"https://api.github.com/users/steverweber/orgs","repos_url":"https://api.github.com/users/steverweber/repos","events_url":"https://api.github.com/users/steverweber/events{/privacy}","received_events_url":"https://api.github.com/users/steverweber/received_events","type":"User","site_admin":false},"created_at":"2015-09-05T17:27:09Z","updated_at":"2015-09-05T17:27:09Z","body":"yes that does hit the master. so your minion can contact the master... but master to minion is broken. I have reviewed the code and so much has changed that its tricky to follow everything... but i'll keep digging for a little longer "}}