{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/28252","labels_url":"https://api.github.com/repos/saltstack/salt/issues/28252/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/28252/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/28252/events","html_url":"https://github.com/saltstack/salt/issues/28252","id":113050720,"number":28252,"title":"Varable defined in outter scpoe is not avaiable in inner scope","user":{"login":"loque1","id":4070276,"avatar_url":"https://avatars.githubusercontent.com/u/4070276?v=3","gravatar_id":"","url":"https://api.github.com/users/loque1","html_url":"https://github.com/loque1","followers_url":"https://api.github.com/users/loque1/followers","following_url":"https://api.github.com/users/loque1/following{/other_user}","gists_url":"https://api.github.com/users/loque1/gists{/gist_id}","starred_url":"https://api.github.com/users/loque1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/loque1/subscriptions","organizations_url":"https://api.github.com/users/loque1/orgs","repos_url":"https://api.github.com/users/loque1/repos","events_url":"https://api.github.com/users/loque1/events{/privacy}","received_events_url":"https://api.github.com/users/loque1/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Core","name":"Core","color":"fef2c0"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P3","name":"P3","color":"0a3d77"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Regression","name":"Regression","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Renderers","name":"Renderers","color":"006b75"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2640,"closed_issues":2813,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-10-26T16:29:56Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-10-23T16:08:22Z","updated_at":"2015-10-26T16:44:11Z","closed_at":null,"body":"Context:\r\n\r\nWe make heavy use of grains and pillars to add meta data about each environment we manage and what data gets passed into that environment. We then use this data to run dynamic configuration of some of our services in most cases its the web server and the load balancer.\r\n\r\nWe have a salt state that dynamically configures a load balancer. \r\nThe key part of the state is the following\r\n``` jinja\r\n{% set balancer_dict = {} %}\r\n{% set services = [] %}\r\n{% for balancer_env in environment_list %}\r\n{% set balancer_list_dict = {} %}\r\n{% set matchexr_prefix = 'G@environment:client_env:' + balancer_env + ' and G@environment:type:' + env_type %}\r\n{% for app in applist.keys() %}\r\n{% set dict_value = salt['publish.publish'](matchexr_prefix+' and G@services:'+app, 'network.get_hostname', expr_form='compound').keys() %}\r\n{% set dict_key = balancer_env+'_'+applist[app] %}\r\n\r\n{% set env_grains_value = salt['publish.publish'](matchexr_prefix, 'grains.items', expr_form='compound').values() %} \r\n{% for grains in env_grains_value -%}\r\n{% for grain, value in grains.iteritems() -%}\r\n{% if grain == 'services' -%}\r\n{% for service in value -%}\r\n{% if service not in services -%}\r\n{% if  services.append(service) -%}{% endif -%}\r\n{% endif -%}\r\n{% endfor -%}\r\n{% endif -%}\r\n{% endfor -%}\r\n{% endfor -%}\r\n{% if balancer_list_dict.update({dict_key.lower():dict_value})%} {% endif %}\r\n{% endfor %}\r\n{% if balancer_dict.update({balancer_env.lower():[balancer_list_dict]})%} {% endif %}\r\n{% endfor %}\r\n```\r\nwhere we have a nested structure to create a dictionary that will be passed in to the configuration template for the load balancer for a particular set of environments. \r\n\r\nThe key line is the following:\r\n```jinja\r\n{% set dict_value = salt['publish.publish'](matchexr_prefix+' and G@services:'+app, 'network.get_hostname', expr_form='compound').keys() %}\r\n```\r\nWhen running this state on \r\n\r\n```\r\n           Salt: 2014.1.4\r\n         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)\r\n         Jinja2: 2.2.1\r\n       M2Crypto: 0.20.2\r\n msgpack-python: 0.1.9.final\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.0.1\r\n         PyYAML: 3.10\r\n          PyZMQ: 2.2.0\r\n            ZMQ: 3.2.1\r\n\r\n```\r\nIt works fine and collects the correct data to pass into the the template configuration file for the load balancer.\r\n\r\nSo far so good. We have a dynamically configured load balancer for all our environments.\r\n\r\nNow when we upgrade to\r\n\r\n```\r\nSalt Version:\r\n           Salt: 2015.8.0\r\n \r\nDependency Versions:\r\n         Jinja2: 2.2.1\r\n       M2Crypto: 0.20.2\r\n           Mako: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.5.0\r\n         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)\r\n           RAET: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n           cffi: Not Installed\r\n       cherrypy: 3.2.2\r\n       dateutil: 1.4.1\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n        libnacl: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: 0.20.3\r\n   python-gnupg: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n \r\nSystem Versions:\r\n           dist: centos 6.7 Final\r\n        machine: x86_64\r\n        release: 2.6.32-431.17.1.el6.x86_64\r\n         system: CentOS 6.7 Final\r\n```\r\n\r\nThe same state fails with the following error.\r\n\r\n```\r\n    Data failed to compile:\r\n----------\r\n    Rendering SLS 'dev:haproxy.test' failed: Jinja variable 'keys' is undefined\r\n```\r\nThis to me seems that the dictionary I should be expecting is not returning and hence can't call the key method.\r\n\r\nAt this point the master configuration file is the same on both servers. We have the peer system turned on \r\n\r\n```yaml\r\npeer:                                                                                                                                                                                                                                     \r\n   .*:\r\n     - network.get_hostname\r\n     - grains.items\r\n     - grains.get\r\n\r\n```\r\nInitially I thought maybe the peer system was broken so we run a test from the the command line on the upgraded salt server.\r\n\r\n```\r\nsalt 'tst-min-001*' cmd.run \"salt-call publish.publish 'G@environment:client_env:HARRIER_QA and G@environment:type:tst and G@services:UPLOADPIPELINE_HOSTS' network.get_hostname  expr_form='compound'\"\r\ntst-min-001.orbitbenefits.capita:\r\n    [INFO    ] Publishing 'network.get_hostname' to tcp://172.23.30.56:4506\r\n    local:\r\n        ----------\r\n        tst-min-001.orbitbenefits.capita:\r\n            tst-min-001\r\n\r\n```\r\nSo the publish.publish module is working and returning the data. So maybe its just the when the module is being run from the state thats causing the issue.\r\n\r\n``` jinja\r\n{% set test = salt['publish.publish']( matchexr_prefix +' and G@services:TCT_HOSTS',  'network.get_hostname', expr_form='compound') %}\r\n\r\n```\r\nseems to return data and compile.\r\n\r\nbut when running the following :\r\n\r\n```\r\n{% for app in applist %}\r\n{% set test = salt['publish.publish']( matchexr_prefix +' and G@services:' + app,  'network.get_hostname', expr_form='compound').keys() %}\r\n{% endfor %}\r\n\r\n```\r\nwe get the same error\r\n\r\n\r\n\r\n```\r\n    Data failed to compile:\r\n----------\r\n    Rendering SLS 'dev:haproxy.test' failed: Jinja variable 'keys' is undefined\r\n```\r\n\r\nits as if the variables from the outer scope are not being passed into the ```{% set %} ``` \r\n\r\nAny help on why this might be happening would be most welcome. This is now blocking the upgrade of salt as we make heavy use of the peer/mine system to dynamically configure some of our environments.\r\n\r\nIf there is anything else that you might need to know please ask.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/151204513","html_url":"https://github.com/saltstack/salt/issues/28252#issuecomment-151204513","issue_url":"https://api.github.com/repos/saltstack/salt/issues/28252","id":151204513,"user":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T16:44:11Z","updated_at":"2015-10-26T16:44:11Z","body":"It's not `app` that's showing up as undefined. Rather, the return from the publish appears to not be coming back as a dictionary. You should do the set without the `.keys()` call and then just insert the result into your template so you can see what that data structure looks like in the debug log."}}