{"action":"created","issue":{"url":"https://api.github.com/repos/grafana/grafana/issues/2209","labels_url":"https://api.github.com/repos/grafana/grafana/issues/2209/labels{/name}","comments_url":"https://api.github.com/repos/grafana/grafana/issues/2209/comments","events_url":"https://api.github.com/repos/grafana/grafana/issues/2209/events","html_url":"https://github.com/grafana/grafana/issues/2209","id":90215515,"number":2209,"title":"building alerting system for grafana","user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/grafana/grafana/labels/comp:%20alerting","name":"comp: alerting","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":57,"created_at":"2015-06-22T21:22:15Z","updated_at":"2015-10-19T22:26:25Z","closed_at":null,"body":"Hi everyone,\r\nI recently joined [raintank](http://www.raintank.io/) and I will be working with @torkelo, @mattttt , and you, on alerting support for Grafana.\r\n\r\nFrom the results of the [Grafana User Survey](https://infogr.am/grafana_user_survey_mar2015) it is obvious that alerting is the most commonly missed feature for Grafana.\r\nI have worked on/with a few alerting systems in the past (nagios, bosun, graph-explorer, etsy's kale stack, ...) and I'm excited about the opportunity in front of us:\r\nwe can take the best of said systems, but combine them with Grafana's focus on a polished user experience, resulting in a powerful alerting system, well-integrated and smooth to work with.\r\n\r\nFirst of all, terminology sync:\r\n\r\n* alerting: executing logic (threshold checks or more advanced) to know the state of an entity. (ok, warning, critical)\r\n* notifications: emails, text messages, posts to chat, etc to make people aware of a state change\r\n* monitoring: this term covers everything about monitoring (data collection, visualizations, alerting) so I won't be using it here.\r\n\r\nI want to spec out requirements, possible implementation ideas and their pro's/cons.  With your feedback, we can adjust, refine and choose a specific direction.\r\n\r\n## General thoughts:\r\n\r\n* integration with existing tools vs built-in: there's some powerfull alerting systems out there (bosun, kale) that deserve integration.\r\nMany alerting systems are more basic (define expression/threshold, get notification when breached), for those it seems integration is not worth the pain (though I won't stop you)  \r\nThe integrations are a long term effort.  I think the low hanging fruit (\"meet 80% of the needs with 20% of the effort\") can be met with a system\r\nthat is more closely tied to Grafana, i.e. compiled into the grafana binary.\r\nThat said, a lot of people confuse seperation of concerns with \"must be different services\".\r\nIf the code is sane, it'll be decoupled packages but there's nothing necessarily wrong with compiling them together. i.e. you could run:\r\n  - 1 grafana binary that does everything (grafana as you know it + all alerting features) for simplicity\r\n  - multiple grafana binaries in different modes (visualization instances and alerting instances) even highly available/redundant setups if you want to, using an external worker queue\r\n\r\nThat said, we don't want to reinvent the wheel: we want alerting code and functionality to integrate well with Grafana, but if high-quality code is compatible, we should use it.  In fact, I have a prototype that leverages some existing bosun code. (see \"Current state\")\r\n\r\n* polling vs stream processing: they have different performance characteristics,\r\nbut they should be able to take the same or similar alerting rule definitions (thresholds, boolean logic, ..), they mostly are about how the actual rules are executed and don't\r\nchange much about how rules are defined.  Since polling is much simpler and should be able to scale fairly far this should IMHO be our initial focus.\r\n\r\n## Current state\r\n\r\nThe [raintank/grafana](https://github.com/raintank/grafana) version currently has an [alerting package](https://github.com/raintank/grafana/tree/master/pkg/alerting)\r\nwith a simple scheduler, an in-process worker bus as well as rabbitmq based, an alert executor and email notifications.\r\nIt uses the bosun expression libraries which gives us the ability to evaluate arbitrarily complex expressions (use several metrics, use boolean logic, math, etc).\r\nThis package is currently raintank-specific but we will merge a generic version of this into upstream grafana.  This will provide an alert execution platform but notably still missing is\r\n1. an interface to create and manage alerting rules\r\n2. state management (acknowledgements etc)\r\n\r\nthese are harder problems, which I hope to tackle with your input.\r\n\r\n## Requirements, Future implementations\r\nFirst off, I think [bosun](http://bosun.org/) is a pretty fantastic system for alerting (not so much for visualization)\r\nYou can make your alerting rules as advanced as you want, and it enables you to fine-tune over time, backtest on historical data, so you can get them just right.\r\nAnd it has a good state machine.\r\nIn theory we could just compile bosun straight into grafana, and leverage bosun via its REST api instead of Golang api, but then we have less finegrained control and\r\nfor now I feel more comfortable trying out piece by piece (piece meaning golang package) and make the integration decision on a case by case basis. Though the integration\r\nmay look different down the road based on experience and as we figure out what we want our alerting to look like.\r\n\r\nEither way, we don't just want great alerting.  We want great alerting combined with great visualizations, notifications with context, and a smooth workflow where you can manage\r\nyour alerts in the same place you manage your visualizations.  So it needs to be nicely integrated into Grafana.  To that end, there's a few things to consider:\r\n\r\n1. some visualized metrics (metrics plotted on graphs) are not alerted on\r\n2. some visualized metrics are alerted on:\r\n * A: with simple threshold checks: easy to visualize alerting logic\r\n * B: with more advanced logic: (e.g. look at standard deviation of the series being plotted, compare current median against historical median, etc): can't easily be visualized nex\r\n     to the input series\r\n3. some metrics used in alerting logic are not to be vizualized\r\n\r\nBasically, there's a bunch of stuff you may want visualized (V), and a bunch of stuff you want alerts (A), and V and A have some overlap.\r\nI need to think about this a bit more and wonder what y'all think.\r\nThere will definitely need to be 1 central place where you can get an overview of all the things you're alerting on, irrespective of where those rules are defined.\r\n\r\nThere's a few more complications which I'll explain through an example sketch of how alerting could look like:\r\n![sketch](https://cloud.githubusercontent.com/assets/20774/8293162/7d1d1fae-18e8-11e5-8f73-6e36e777b790.png)\r\n\r\n\r\nlet's say we have a timeseries for requests (A) and one for errorous requests (B) and this is what we want to plot.\r\nwe then use fields C,D,E to put stuff that we don't want to alert on.\r\nC contains the formula for ratio of error requests against the total.\r\n\r\nwe may for example want to alert (see E) if the median of this ratio in the last 5min ago is more than 1.5 of what the ratio was in the same 5minute period last week, and also\r\nif the errors seen in the last 5min is worse than the errors seen since 2 months ago until 5min ago.\r\n\r\nnotes:\r\n* some queries use different timeranges than what is rendered\r\n* in addition to processing by tsdb (such as Graphite's sum(), divide() etc which return series) we need to be able to reduce series to single numbers. fairly easy to implement (and in fact currently the bosun library does this for us)\r\n* we need boolean logic (bosun also gives us this)\r\n* in this example the expression only uses variables defined within the same panel, but it might make sense to include expressions of other panels/graphs.\r\n\r\nother ponderings:\r\n* do we integrate with current grafana graph threshold settings (which are currently for viz only, not for processing) ? if the expression is a threshold check, we could automatically\r\ndisplay a threshold line\r\n* using the letters is a bit clunky, could we refer to the aliases instead? like #requests and #errors?\r\n* if the expression are `stats.$site.requests` and `stats.$site.errors`, and we want to have seperate alert instances for every site (but only set up the rule once)? what if we only want it for a select few of the sites. what if we want different parameters based on which site? bosun actually supports all these features, and we could expose them though we should probably build a UI around them.\r\n\r\nI think for an initial implementation every graph could have two fields, like so:\r\n```\r\nwarn: - expression\r\n         - notification settings (email,http hook, ..)\r\ncrit: - expression\r\n        -notification settings\r\n```\r\nwhere the expression is something like what I put in E in the sketch.\r\nfor logic/data that we don't want to visualize, we just toggle off the visibility icon.\r\ngrafana would replace the variables in the formula's, execute the expression (with the current bosun based executor). results (state changes) could be fed into something like elasticsearch and displayed via the annotations system.\r\n\r\nThoughts?\r\nDo you have concerns or needs that I didn't addres?\r\n"},"comment":{"url":"https://api.github.com/repos/grafana/grafana/issues/comments/149365122","html_url":"https://github.com/grafana/grafana/issues/2209#issuecomment-149365122","issue_url":"https://api.github.com/repos/grafana/grafana/issues/2209","id":149365122,"user":{"login":"jaimegago","id":1704368,"avatar_url":"https://avatars.githubusercontent.com/u/1704368?v=3","gravatar_id":"","url":"https://api.github.com/users/jaimegago","html_url":"https://github.com/jaimegago","followers_url":"https://api.github.com/users/jaimegago/followers","following_url":"https://api.github.com/users/jaimegago/following{/other_user}","gists_url":"https://api.github.com/users/jaimegago/gists{/gist_id}","starred_url":"https://api.github.com/users/jaimegago/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaimegago/subscriptions","organizations_url":"https://api.github.com/users/jaimegago/orgs","repos_url":"https://api.github.com/users/jaimegago/repos","events_url":"https://api.github.com/users/jaimegago/events{/privacy}","received_events_url":"https://api.github.com/users/jaimegago/received_events","type":"User","site_admin":false},"created_at":"2015-10-19T22:26:25Z","updated_at":"2015-10-19T22:26:25Z","body":"@mattttt @Dieterbe that's great ! Looks like you're on the path of \"batteries included but removable\" which is IMHO the best of both worlds. Have you already thought about how you are going to pass authorization data to the alert handler? I'm thinking a story like this:\r\nAs a Grafana user I'd like to be alerted via <email> or <pagerduty> or <foo> when <some condition built via Grafana alerting UI>.\r\nThat user should only be able to send alerts to the notifications system he is authorized for, this is a requirement for self service and will need to be addressed someway somehow. Since Grafana 2 we have a SQL backend + users authentication/authorization with LDAP integration, so it doesn't seem too far fetch to have that capability from day one of alerting?\r\nWith Sensu (which is the tool I'd be plugging in) passing the alert target (e.g. email address) via the handler should be quite straight forward, can't say about the others.\r\n\r\n"}}