{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1128","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1128/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1128/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1128/events","html_url":"https://github.com/prometheus/prometheus/issues/1128","id":108887210,"number":1128,"title":"Alert template expansion is broken for global labels","user":{"login":"dan-cleinmark","id":3299293,"avatar_url":"https://avatars.githubusercontent.com/u/3299293?v=3","gravatar_id":"","url":"https://api.github.com/users/dan-cleinmark","html_url":"https://github.com/dan-cleinmark","followers_url":"https://api.github.com/users/dan-cleinmark/followers","following_url":"https://api.github.com/users/dan-cleinmark/following{/other_user}","gists_url":"https://api.github.com/users/dan-cleinmark/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-cleinmark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-cleinmark/subscriptions","organizations_url":"https://api.github.com/users/dan-cleinmark/orgs","repos_url":"https://api.github.com/users/dan-cleinmark/repos","events_url":"https://api.github.com/users/dan-cleinmark/events{/privacy}","received_events_url":"https://api.github.com/users/dan-cleinmark/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-29T14:53:15Z","updated_at":"2015-09-29T15:02:14Z","closed_at":null,"body":"Per @juliusv on IRC: \"the template expansion happens before the global labels are attached: https://github.com/prometheus/prometheus/blob/master/rules/manager.go#L221-L222\".  This results in references to global labels to not expand in alerts.\r\n\r\nRelevant bits from an email alert:\r\n\r\n    Subject: [ALERT] Instance_prometheus_node_exporter_is_broken: <no value>/xxx_node is BROKEN!\r\n\r\n    <no value>: prometheus xxx_node exporter is broken\r\n\r\n    Alertmanager: http://xxx:9093/\r\n    Runbook entry: < link >\r\n\r\n    Grouping labels:\r\n\r\n      alertname = \"Instance_prometheus_node_exporter_is_broken\"\r\n      domain = \"prometheus\"\r\n      env = \"ci\"\r\n      instance = \"xxx_node\"\r\n      job = \"xxx\"\r\n      service = \"xxx\"\r\n      severity = \"email\"\r\n\r\n    Payload labels:\r\n\r\n      activeSince = \"2015-09-28T23:59:40.37Z\"\r\n      alertingRule = \"ALERT Instance_prometheus_node_exporter_is_broken\r\n    \tIF up{job=~\"^.+_node$\"} == 0\r\n    \tFOR 5m\r\n    \tWITH {domain=\"prometheus\", severity=\"email\"}\r\n    \tSUMMARY \"{{$labels.env}}/{{$labels.job}} is BROKEN!\"\r\n    \tDESCRIPTION \"{{$labels.env}}: prometheus {{$labels.job}} exporter is broken\"\r\n    \tRUNBOOK \"< link >\"\r\n      generatorURL = \"http://xxx:9090/graph#%5B%7B%22expr%22%3A%22up%7Bjob%3D~%5C%22%5E.%2B_node%24%5C%22%7D%20%3D%3D%200%22%2C%22tab%22%3A0%7D%5D\"\r\n      value = \"0\""},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/144087341","html_url":"https://github.com/prometheus/prometheus/issues/1128#issuecomment-144087341","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1128","id":144087341,"user":{"login":"juliusv","id":538008,"avatar_url":"https://avatars.githubusercontent.com/u/538008?v=3","gravatar_id":"","url":"https://api.github.com/users/juliusv","html_url":"https://github.com/juliusv","followers_url":"https://api.github.com/users/juliusv/followers","following_url":"https://api.github.com/users/juliusv/following{/other_user}","gists_url":"https://api.github.com/users/juliusv/gists{/gist_id}","starred_url":"https://api.github.com/users/juliusv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juliusv/subscriptions","organizations_url":"https://api.github.com/users/juliusv/orgs","repos_url":"https://api.github.com/users/juliusv/repos","events_url":"https://api.github.com/users/juliusv/events{/privacy}","received_events_url":"https://api.github.com/users/juliusv/received_events","type":"User","site_admin":false},"created_at":"2015-09-29T15:02:14Z","updated_at":"2015-09-29T15:02:14Z","body":"@brian-brazil Agreed that this use case would better be solved via notification templates on the Alertmanager side. However, is there harm in also setting the global labels here? In the worst case, someone uses them where they weren't able to use them before..."}}