{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/20240","labels_url":"https://api.github.com/repos/saltstack/salt/issues/20240/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/20240/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/20240/events","html_url":"https://github.com/saltstack/salt/issues/20240","id":56009902,"number":20240,"title":"Minion did not return","user":{"login":"Jiaion","id":451960,"avatar_url":"https://avatars.githubusercontent.com/u/451960?v=3","gravatar_id":"","url":"https://api.github.com/users/Jiaion","html_url":"https://github.com/Jiaion","followers_url":"https://api.github.com/users/Jiaion/followers","following_url":"https://api.github.com/users/Jiaion/following{/other_user}","gists_url":"https://api.github.com/users/Jiaion/gists{/gist_id}","starred_url":"https://api.github.com/users/Jiaion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jiaion/subscriptions","organizations_url":"https://api.github.com/users/Jiaion/orgs","repos_url":"https://api.github.com/users/Jiaion/repos","events_url":"https://api.github.com/users/Jiaion/events{/privacy}","received_events_url":"https://api.github.com/users/Jiaion/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"pitatus","id":9560004,"avatar_url":"https://avatars.githubusercontent.com/u/9560004?v=3","gravatar_id":"","url":"https://api.github.com/users/pitatus","html_url":"https://github.com/pitatus","followers_url":"https://api.github.com/users/pitatus/followers","following_url":"https://api.github.com/users/pitatus/following{/other_user}","gists_url":"https://api.github.com/users/pitatus/gists{/gist_id}","starred_url":"https://api.github.com/users/pitatus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pitatus/subscriptions","organizations_url":"https://api.github.com/users/pitatus/orgs","repos_url":"https://api.github.com/users/pitatus/repos","events_url":"https://api.github.com/users/pitatus/events{/privacy}","received_events_url":"https://api.github.com/users/pitatus/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":205,"closed_issues":421,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2015-02-04T21:58:01Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-01-30T09:07:48Z","updated_at":"2015-02-05T09:31:47Z","closed_at":null,"body":"#Hi ：\r\n   I found salt minion 2014.7 very unstable, often not return。\r\n   I use 2014.7 commit id 3e2b366cd6f8c8bcb37eb55f59853178a0420072\r\n\r\n````\r\n#salt 'console1*' test.ping -v \r\nExecuting job with jid 20150130165611866138\r\n-------------------------------------------\r\n\r\nconsole1.xxx:\r\n    Minion did not return. [No response]\r\n````\r\n\r\nrun command  return null\r\n#salt-run jobs.lookup_jid 20150130165611866138 \r\n\r\n# Minion debug log \r\n\r\n```\r\n2015-01-30 16:56:12,695 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:927 User root Executing command test.ping with jid 20150130165611866138\r\n2015-01-30 16:56:12,696 [salt.minion                                 ][DEBUG   ] /usr/local/salt/packages/salt/minion.py:933 Command details {'tgt_type': 'glob', 'jid': '20150130165611866138', 'tgt': 'console1*', 'ret': '', 'user': 'root', 'arg': [], 'fun': 'test.ping'}\r\n2015-01-30 16:56:12,707 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:1006 Starting a new job with PID 15879\r\n2015-01-30 16:56:12,708 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:1174 Returning information for job: 20150130165611866138\r\n2015-01-30 16:56:12,841 [salt.crypt                                  ][DEBUG   ] /usr/local/salt/packages/salt/crypt.py:399 Decrypting the current master AES key\r\n2015-01-30 16:56:12,842 [salt.crypt                                  ][DEBUG   ] /usr/local/salt/packages/salt/crypt.py:324 Loaded minion key: /etc/salt/pki/minion/minion.pem\r\n2015-01-30 16:56:17,715 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:927 User root Executing command saltutil.find_job with jid 20150130165616879623\r\n2015-01-30 16:56:17,716 [salt.minion                                 ][DEBUG   ] /usr/local/salt/packages/salt/minion.py:933 Command details {'tgt_type': 'glob', 'jid': '20150130165616879623', 'tgt': 'console1*', 'ret': '', 'user': 'root', 'arg': ['20150130165611866138'], 'fun': 'saltutil.find_job'}\r\n2015-01-30 16:56:17,726 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:1006 Starting a new job with PID 15961\r\n2015-01-30 16:56:17,730 [salt.minion                                 ][INFO    ] /usr/local/salt/packages/salt/minion.py:1174 Returning information for job: 20150130165616879623\r\n2015-01-30 16:56:17,860 [salt.crypt                                  ][DEBUG   ] /usr/local/salt/packages/salt/crypt.py:399 Decrypting the current master AES key\r\n2015-01-30 16:56:17,861 [salt.crypt                                  ][DEBUG   ] /usr/local/salt/packages/salt/crypt.py:324 Loaded minion key: /etc/salt/pki/minion/minion.pem\r\n```\r\n\r\n# Minion error log \r\n\r\n```\r\n#less /var/log/salt/minion                             \r\n2015-01-30 14:39:28,359 [salt.minion                                 ][CRITICAL] /usr/local/salt/packages/salt/minion.py:1734 An exception occurred while polling the minion\r\nTraceback (most recent call last):\r\n  File \"/usr/local/salt/packages/salt/minion.py\", line 1726, in tune_in_no_block\r\n    self._do_socket_recv(socks)\r\n  File \"/usr/local/salt/packages/salt/minion.py\", line 1760, in _do_socket_recv\r\n    self._handle_payload(payload)\r\n  File \"/usr/local/salt/packages/salt/minion.py\", line 866, in _handle_payload\r\n    payload['sig'] if 'sig' in payload else None)\r\n  File \"/usr/local/salt/packages/salt/minion.py\", line 897, in _handle_aes\r\n    data = self.crypticle.loads(load)\r\n  File \"/usr/local/salt/packages/salt/crypt.py\", line 791, in loads\r\n    data = self.decrypt(data)\r\n  File \"/usr/local/salt/packages/salt/crypt.py\", line 774, in decrypt\r\n    raise AuthenticationError('message authentication failed')\r\nAuthenticationError: message authentication failed\r\n```\r\n\r\n```\r\n           Salt: 2014.7.0\r\n         Python: 2.6.6 (r266:84292, Sep 12 2011, 14:03:14)\r\n         Jinja2: 2.8-dev\r\n       M2Crypto: 0.21.1\r\n msgpack-python: 0.4.0\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.6.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.10\r\n          ioflo: Not Installed\r\n          PyZMQ: 13.0.2\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.2\r\n           Mako: 0.9.0\r\n```\r\n\r\n# But very fast 0.17.5\r\n\r\n```\r\n#time salt '*' test.ping -v \r\nExecuting job with jid 20150130161206249724\r\n-------------------------------------------\r\nlvs.xxx:\r\n    True\r\ncache.xxx:\r\n    True\r\nweb.xxx:\r\n    True\r\nlog.xxx:\r\n    True\r\n\r\nreal    0m0.666s\r\nuser    0m0.422s\r\nsys     0m0.044s\r\n```\r\n```\r\n           Salt: 0.17.5\r\n         Python: 2.6.6 (r266:84292, Sep 12 2011, 14:03:14)\r\n         Jinja2: 2.8-dev\r\n       M2Crypto: 0.21.1\r\n msgpack-python: 0.4.0\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.6.1\r\n         PyYAML: 3.10\r\n          PyZMQ: 13.0.2\r\n            ZMQ: 3.2.2\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/73017871","html_url":"https://github.com/saltstack/salt/issues/20240#issuecomment-73017871","issue_url":"https://api.github.com/repos/saltstack/salt/issues/20240","id":73017871,"user":{"login":"andrejohansson","id":2828428,"avatar_url":"https://avatars.githubusercontent.com/u/2828428?v=3","gravatar_id":"","url":"https://api.github.com/users/andrejohansson","html_url":"https://github.com/andrejohansson","followers_url":"https://api.github.com/users/andrejohansson/followers","following_url":"https://api.github.com/users/andrejohansson/following{/other_user}","gists_url":"https://api.github.com/users/andrejohansson/gists{/gist_id}","starred_url":"https://api.github.com/users/andrejohansson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrejohansson/subscriptions","organizations_url":"https://api.github.com/users/andrejohansson/orgs","repos_url":"https://api.github.com/users/andrejohansson/repos","events_url":"https://api.github.com/users/andrejohansson/events{/privacy}","received_events_url":"https://api.github.com/users/andrejohansson/received_events","type":"User","site_admin":false},"created_at":"2015-02-05T09:31:47Z","updated_at":"2015-02-05T09:31:47Z","body":"I've been running things in Azure (both master and minions) and had these problems as well. At first I put up a cron that pingeded every minute (Azure seemed to cap ovar 60s idle connections) and it worked at first. But when the number of minions grew this solution just spammed us down.\r\n\r\nI've got a test setup running with one master and one minion now, and activated the keepalive settings with intervals less than 60s on the minion. This seems to work (2014.7.1 minion, master from dev branch). \r\n\r\nSo far this seems to work! However...efter a long idle period the first ping takes 10s, the rest takes 1s ...but thats another issue I think. The connection seems to be there!\r\n\r\n"}}