{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/20777","labels_url":"https://api.github.com/repos/saltstack/salt/issues/20777/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/20777/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/20777/events","html_url":"https://github.com/saltstack/salt/issues/20777","id":57996003,"number":20777,"title":"2014.7.0 master could not control 2014.7.1 minion","user":{"login":"wt","id":416255,"avatar_url":"https://avatars.githubusercontent.com/u/416255?v=3","gravatar_id":"","url":"https://api.github.com/users/wt","html_url":"https://github.com/wt","followers_url":"https://api.github.com/users/wt/followers","following_url":"https://api.github.com/users/wt/following{/other_user}","gists_url":"https://api.github.com/users/wt/gists{/gist_id}","starred_url":"https://api.github.com/users/wt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wt/subscriptions","organizations_url":"https://api.github.com/users/wt/orgs","repos_url":"https://api.github.com/users/wt/repos","events_url":"https://api.github.com/users/wt/events{/privacy}","received_events_url":"https://api.github.com/users/wt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2050,"closed_issues":1690,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-03-19T05:38:50Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-02-17T22:58:13Z","updated_at":"2015-03-19T07:44:37Z","closed_at":null,"body":"Current \"salt --versions\":\r\n```\r\n           Salt: 2014.7.1\r\n         Python: 2.6.9 (unknown, Sep 13 2014, 00:25:11)\r\n         Jinja2: 2.7.2\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.1.13\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.6.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.10\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.3.1\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.4\r\n           Mako: Not Installed\r\n```\r\n\r\nI use epel6 packages. Over the weekend, the packages were upgraded in the epel6 repo. I have a pkg.latest that installs the salt-master and salt-minions on my fleet.\r\n\r\nWhen the master upgraded, the init script failed to stop the 2014.7.0 master. As a result that 2014.7.0 master was left running after the package was installed (and also after a cronjob that did \"service salt-master restart\" had run and after a manual attempt to \"service salt-master restart\").\r\n\r\nSimultaneously, some minions were upgraded to 2014.7.0. I don't know if the package restarted them, but they were successfully restarted by a cron job. When the 2014.7.1 minions were attempting to run scheduled highstates, I would see the following in the logs:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/schedule.py\", line 437, in handle_func\r\n    ret['return'] = self.functions[func]()\r\n  File \"/usr/lib/python2.6/site-packages/salt/modules/state.py\", line 325, in highstate\r\n    whitelist=kwargs.get('whitelist')\r\n  File \"/usr/lib/python2.6/site-packages/salt/state.py\", line 2849, in call_highstate\r\n    self.load_dynamic(matches)\r\n  File \"/usr/lib/python2.6/site-packages/salt/state.py\", line 2394, in load_dynamic\r\n    refresh=False)\r\n  File \"/usr/lib/python2.6/site-packages/salt/modules/saltutil.py\", line 388, in sync_all\r\n    ret['states'] = sync_states(saltenv, False)\r\n  File \"/usr/lib/python2.6/site-packages/salt/modules/saltutil.py\", line 269, in sync_states\r\n    ret = _sync('states', saltenv)\r\n  File \"/usr/lib/python2.6/site-packages/salt/modules/saltutil.py\", line 98, in _sync\r\n    source, sub_env, include_pat=r'E@\\.(pyx?|so)$'\r\n  File \"/usr/lib/python2.6/site-packages/salt/modules/cp.py\", line 416, in cache_dir\r\n    path, saltenv, include_empty, include_pat, exclude_pat\r\n  File \"/usr/lib/python2.6/site-packages/salt/fileclient.py\", line 224, in cache_dir\r\n    fn_ = self.cache_file('salt://' + fn_, saltenv)\r\n  File \"/usr/lib/python2.6/site-packages/salt/fileclient.py\", line 147, in cache_file\r\n    return self.get_url(path, '', True, saltenv)\r\n  File \"/usr/lib/python2.6/site-packages/salt/fileclient.py\", line 521, in get_url\r\n    return self.get_file(url, dest, makedirs, saltenv)\r\n  File \"/usr/lib/python2.6/site-packages/salt/fileclient.py\", line 991, in get_file\r\n    if not data['data']:\r\nTypeError: string indices must be integers, not str\r\n```\r\n\r\nOnce I manually killed the old masters (SIGKILL to the pids this time) and restarted the master, the minions started to work. I am sorry, but I don't have any more info.\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/83395318","html_url":"https://github.com/saltstack/salt/issues/20777#issuecomment-83395318","issue_url":"https://api.github.com/repos/saltstack/salt/issues/20777","id":83395318,"user":{"login":"bersace","id":542613,"avatar_url":"https://avatars.githubusercontent.com/u/542613?v=3","gravatar_id":"","url":"https://api.github.com/users/bersace","html_url":"https://github.com/bersace","followers_url":"https://api.github.com/users/bersace/followers","following_url":"https://api.github.com/users/bersace/following{/other_user}","gists_url":"https://api.github.com/users/bersace/gists{/gist_id}","starred_url":"https://api.github.com/users/bersace/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bersace/subscriptions","organizations_url":"https://api.github.com/users/bersace/orgs","repos_url":"https://api.github.com/users/bersace/repos","events_url":"https://api.github.com/users/bersace/events{/privacy}","received_events_url":"https://api.github.com/users/bersace/received_events","type":"User","site_admin":false},"created_at":"2015-03-19T07:44:37Z","updated_at":"2015-03-19T07:44:37Z","body":"Hi,\r\n\r\nI confirm this bug with same versions :\r\n\r\n```console\r\n# salt-master --versions-report\r\n           Salt: 2014.7.1\r\n         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)\r\n         Jinja2: 2.7.3\r\n       M2Crypto: 0.21.1\r\n msgpack-python: 0.4.2\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.6.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.11\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.4.0\r\n           RAET: Not Installed\r\n            ZMQ: 4.0.5\r\n           Mako: 1.0.0\r\n# salt-minion --versions-report\r\n           Salt: 2014.7.1\r\n         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)\r\n         Jinja2: 2.7.3\r\n       M2Crypto: 0.21.1\r\n msgpack-python: 0.4.2\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.6.1\r\n        libnacl: Not Installed\r\n         PyYAML: 3.11\r\n          ioflo: Not Installed\r\n          PyZMQ: 14.4.0\r\n           RAET: Not Installed\r\n            ZMQ: 4.0.5\r\n           Mako: 1.0.0\r\n```"}}