{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/22650","labels_url":"https://api.github.com/repos/saltstack/salt/issues/22650/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/22650/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/22650/events","html_url":"https://github.com/saltstack/salt/issues/22650","id":68373723,"number":22650,"title":"Random 'Failed to authenticate!' while running commands on master.","user":{"login":"jakubmm","id":11942157,"avatar_url":"https://avatars.githubusercontent.com/u/11942157?v=3","gravatar_id":"","url":"https://api.github.com/users/jakubmm","html_url":"https://github.com/jakubmm","followers_url":"https://api.github.com/users/jakubmm/followers","following_url":"https://api.github.com/users/jakubmm/following{/other_user}","gists_url":"https://api.github.com/users/jakubmm/gists{/gist_id}","starred_url":"https://api.github.com/users/jakubmm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakubmm/subscriptions","organizations_url":"https://api.github.com/users/jakubmm/orgs","repos_url":"https://api.github.com/users/jakubmm/repos","events_url":"https://api.github.com/users/jakubmm/events{/privacy}","received_events_url":"https://api.github.com/users/jakubmm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","html_url":"https://github.com/saltstack/salt/milestones/Blocked","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":247,"closed_issues":497,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2015-04-17T15:04:17Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-04-14T13:27:18Z","updated_at":"2015-04-17T15:50:52Z","closed_at":null,"body":"We're getting random 'Failed to authenticate' messages when running commands on salt master, and occasionally minions are TypeErroring:\r\n```\r\n[j@salt-conf salt]$ sudo salt -v '008*' test.ping\r\nExecuting job with jid 20150414123557337413\r\n-------------------------------------------\r\n\r\nFailed to authenticate!  This is most likely because this user is not permitted to execute commands, but there is a small possibility that a disk error occurred (check disk/inode usage).\r\n[j@salt-conf salt]$\r\n```\r\nVersions on the master:\r\n```\r\n[j@salt-conf salt]$ salt-master --versions\r\n           Salt: 2014.7.1\r\n         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)\r\n         Jinja2: unknown\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.4.4\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.0.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.10\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.3.1\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.5\r\n           Mako: Not Installed\r\n[j@salt-conf salt]$ \r\n```\r\nVersions on the minion:\r\n```\r\n           Salt: 2014.7.0\r\n         Python: 2.6.8 (unknown, Nov  7 2012, 14:47:45)\r\n         Jinja2: 2.5.5\r\n       M2Crypto: 0.21.1\r\n msgpack-python: 0.1.12\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.3\r\n        libnacl: Not Installed\r\n         PyYAML: 3.08\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.3.1\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.2\r\n           Mako: Not Installed\r\n```\r\n/var/log/salt/minion on salt minion:\r\n```\r\n2015-04-14 13:43:46,223 [salt.minion                                 ][INFO    ] User sudo_j Executing command saltutil.refresh_pillar with jid 20150414134346225534\r\n2015-04-14 13:43:46,235 [salt.minion                                 ][INFO    ] Starting a new job with PID 12093\r\n2015-04-14 13:43:46,265 [salt.minion                                 ][INFO    ] Returning information for job: 20150414134346225534\r\n2015-04-14 13:51:04,019 [salt             ][INFO    ] Setting up the Salt Minion \"008.domain.ie\"\r\n2015-04-14 13:51:04,760 [salt.minion                              ][INFO    ] Authentication with master at 172.30.143.6 successful!\r\n2015-04-14 13:51:05,093 [salt.log.setup                           ][ERROR   ] An un-handled exception was caught by salt's global exception handler:\r\nTypeError: string indices must be integers, not str\r\nTraceback (most recent call last):\r\n  File \"/usr/bin/salt-minion\", line 14, in <module>\r\n    salt_minion()\r\n  File \"/usr/lib/python2.6/site-packages/salt/scripts.py\", line 56, in salt_minion\r\n    minion.start()\r\n  File \"/usr/lib/python2.6/site-packages/salt/__init__.py\", line 262, in start\r\n    self.prepare()\r\n  File \"/usr/lib/python2.6/site-packages/salt/__init__.py\", line 241, in prepare\r\n    self.minion = salt.minion.Minion(self.config)\r\n  File \"/usr/lib/python2.6/site-packages/salt/minion.py\", line 603, in __init__\r\n    opts['environment'],\r\n  File \"/usr/lib/python2.6/site-packages/salt/pillar/__init__.py\", line 91, in compile_pillar\r\n    ret_pillar = self.sreq.crypted_transfer_decode_dictentry(load, dictkey='pillar', tries=3, timeout=7200)\r\n  File \"/usr/lib/python2.6/site-packages/salt/transport/__init__.py\", line 236, in crypted_transfer_decode_dictentry\r\n    aes = key.private_decrypt(ret['key'], 4)\r\nTypeError: string indices must be integers, not str\r\n```\r\nAny hints how to pin down the root cause will be appreciated!\r\n\r\nRegards,\r\nJakub"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/94013728","html_url":"https://github.com/saltstack/salt/issues/22650#issuecomment-94013728","issue_url":"https://api.github.com/repos/saltstack/salt/issues/22650","id":94013728,"user":{"login":"rallytime","id":6547906,"avatar_url":"https://avatars.githubusercontent.com/u/6547906?v=3","gravatar_id":"","url":"https://api.github.com/users/rallytime","html_url":"https://github.com/rallytime","followers_url":"https://api.github.com/users/rallytime/followers","following_url":"https://api.github.com/users/rallytime/following{/other_user}","gists_url":"https://api.github.com/users/rallytime/gists{/gist_id}","starred_url":"https://api.github.com/users/rallytime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rallytime/subscriptions","organizations_url":"https://api.github.com/users/rallytime/orgs","repos_url":"https://api.github.com/users/rallytime/repos","events_url":"https://api.github.com/users/rallytime/events{/privacy}","received_events_url":"https://api.github.com/users/rallytime/received_events","type":"User","site_admin":false},"created_at":"2015-04-17T15:50:52Z","updated_at":"2015-04-17T15:50:52Z","body":"Definitely! Thanks very much for all of this information. One thing I would suggest for sure is to upgrade your versions of ZMQ on your master and minions to see if that helps reduce some of these spurious errors. There are squirrelly issues like this that have been fixed in the past by doing that as there are known bugs with 3.x versions of ZMQ.\r\n\r\nZMQ 4 can be found for CentOS 5 in our [COPR repo](https://copr.fedoraproject.org/coprs/saltstack/salt-el5/)"}}