{"action":"opened","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/29465","labels_url":"https://api.github.com/repos/saltstack/salt/issues/29465/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/29465/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/29465/events","html_url":"https://github.com/saltstack/salt/issues/29465","id":120604365,"number":29465,"title":"Orchestrate runner (or maybe saltmod.state?) wrongly reports success when minion is not connected","user":{"login":"jefferyharrell","id":8659114,"avatar_url":"https://avatars.githubusercontent.com/u/8659114?v=3","gravatar_id":"","url":"https://api.github.com/users/jefferyharrell","html_url":"https://github.com/jefferyharrell","followers_url":"https://api.github.com/users/jefferyharrell/followers","following_url":"https://api.github.com/users/jefferyharrell/following{/other_user}","gists_url":"https://api.github.com/users/jefferyharrell/gists{/gist_id}","starred_url":"https://api.github.com/users/jefferyharrell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jefferyharrell/subscriptions","organizations_url":"https://api.github.com/users/jefferyharrell/orgs","repos_url":"https://api.github.com/users/jefferyharrell/repos","events_url":"https://api.github.com/users/jefferyharrell/events{/privacy}","received_events_url":"https://api.github.com/users/jefferyharrell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-06T03:14:57Z","updated_at":"2015-12-06T03:14:57Z","closed_at":null,"body":"Start with a trivial file, which I called `state/whatever/init.sls`:\r\n\r\n```salt\r\nsuccess:\r\n  test.nop: []\r\n```\r\n\r\nExecute this state on some rando:\r\n\r\n```\r\n[admin@bonnie ~]$ sudo salt milo.dallas.charlietango.com state.sls state.whatever\r\nmilo.dallas.charlietango.com:\r\n----------\r\n          ID: success\r\n    Function: test.nop\r\n      Result: True\r\n     Comment: Success!\r\n     Started: 20:45:06.042724\r\n    Duration: 0.81 ms\r\n     Changes:   \r\n\r\nSummary for milo.dallas.charlietango.com\r\n------------\r\nSucceeded: 1\r\nFailed:    0\r\n------------\r\nTotal states run:     1\r\nTotal run time:   0.810 ms\r\n```\r\n\r\nNow execute it on a minion that's not currently responding. To simulate this I just stop the `salt-minion` service on my minion with `systemctl stop salt-minion`. After a brief timeout:\r\n\r\n```\r\n[admin@bonnie ~]$ sudo salt milo.dallas.charlietango.com state.sls state.whatever\r\nmilo.dallas.charlietango.com:\r\n    Minion did not return. [Not connected]\r\n```\r\n\r\nSo far so good. Now create an orchestration that invokes this state. Here's mine:\r\n\r\n```salt\r\ntestcase1:\r\n  salt.state:\r\n    - tgt: milo.dallas.charlietango.com\r\n    - sls:\r\n      - state.whatever\r\n```\r\n\r\nWith `salt-minion` running as normal on Milo, run the orchestration, and we get:\r\n\r\n```\r\n[admin@bonnie ~]$ sudo salt-run state.orchestrate orchestrate.testcase\r\nbonnie.dallas.charlietango.com_master:\r\n----------\r\n          ID: testcase1\r\n    Function: salt.state\r\n      Result: True\r\n     Comment: States ran successfully. No changes made to milo.dallas.charlietango.com.\r\n     Started: 20:49:59.247537\r\n    Duration: 366.16 ms\r\n     Changes:   \r\n\r\nSummary for bonnie.dallas.charlietango.com_master\r\n------------\r\nSucceeded: 1\r\nFailed:    0\r\n------------\r\nTotal states run:     1\r\nTotal run time: 366.160 ms\r\n```\r\n\r\nNow stop the `salt-minion` on Milo and rerun the orchestration and we get, after the expected timeout, this surprising result:\r\n\r\n```\r\n[admin@bonnie ~]$ sudo salt-run state.orchestrate orchestrate.testcase\r\nbonnie.dallas.charlietango.com_master:\r\n----------\r\n          ID: testcase1\r\n    Function: salt.state\r\n      Result: True\r\n     Comment: States ran successfully.\r\n     Started: 20:50:33.358693\r\n    Duration: 15099.234 ms\r\n     Changes:   \r\n\r\nSummary for bonnie.dallas.charlietango.com_master\r\n------------\r\nSucceeded: 1\r\nFailed:    0\r\n------------\r\nTotal states run:     1\r\nTotal run time:  15.099 s\r\n```\r\n\r\nThings get ugly, in my opinion, when you do this:\r\n\r\n```salt\r\ntestcase1:\r\n  salt.state:\r\n    - tgt: milo.dallas.charlietango.com\r\n    - sls:\r\n      - state.whatever\r\n\r\ntestcase2:\r\n  salt.function:\r\n    - name: test.true\r\n    - tgt: bonnie.dallas.charlietango.com\r\n    - onfail:\r\n      - salt: testcase1\r\n```\r\n\r\nWhen Milo is not connected, we get this entirely unexpected result:\r\n\r\n```\r\n[admin@bonnie ~]$ sudo salt-run state.orchestrate orchestrate.testcase\r\nbonnie.dallas.charlietango.com_master:\r\n----------\r\n          ID: testcase1\r\n    Function: salt.state\r\n      Result: True\r\n     Comment: States ran successfully.\r\n     Started: 21:01:50.644215\r\n    Duration: 15099.056 ms\r\n     Changes:   \r\n----------\r\n          ID: testcase2\r\n    Function: salt.function\r\n        Name: test.true\r\n      Result: True\r\n     Comment: State was not run because onfail req did not change\r\n     Started: \r\n    Duration: \r\n     Changes:   \r\n\r\nSummary for bonnie.dallas.charlietango.com_master\r\n------------\r\nSucceeded: 2\r\nFailed:    0\r\n------------\r\nTotal states run:     2\r\nTotal run time:  15.099 s\r\n```\r\n\r\nAs you can see from the run time at the bottom, Milo really wasn't responding when I ran that last orchestration. The master timed out waiting for it. But the master didn't recognize that as a failure, so it didn't trigger the `onfail` requisite in `testcase2`. To the guy sitting at the keyboard, it looks as if the orchestration succeeded just fine, although taking abnormally long.\r\n\r\nHere are my version reports for master and minion:\r\n\r\n```\r\n[admin@bonnie ~]$ salt-master --versions-report\r\nSalt Version:\r\n           Salt: 2015.8.3\r\n \r\nDependency Versions:\r\n         Jinja2: 2.7.2\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)\r\n           RAET: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n           cffi: 0.8.6\r\n       cherrypy: 3.2.2\r\n       dateutil: Not Installed\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n        libnacl: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: 2.10\r\n       pycrypto: 2.6.1\r\n         pygit2: 0.21.4\r\n   python-gnupg: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n \r\nSystem Versions:\r\n           dist: centos 7.1.1503 Core\r\n        machine: x86_64\r\n        release: 3.10.0-229.14.1.el7.x86_64\r\n         system: CentOS Linux 7.1.1503 Core\r\n```\r\n\r\n```\r\n[admin@milo ~]$ salt-minion --versions-report\r\nSalt Version:\r\n           Salt: 2015.8.3\r\n \r\nDependency Versions:\r\n         Jinja2: 2.7.2\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)\r\n           RAET: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: Not Installed\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n        libnacl: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n   python-gnupg: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n \r\nSystem Versions:\r\n           dist: centos 7.1.1503 Core\r\n        machine: x86_64\r\n        release: 3.10.0-229.20.1.el7.x86_64\r\n         system: CentOS Linux 7.1.1503 Core\r\n```\r\n\r\nThanks as always."}}