{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1161","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1161/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1161/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1161/events","html_url":"https://github.com/prometheus/prometheus/pull/1161","id":110806099,"number":1161,"title":"promql: Remove extrapolation from rate/increase/delta.","user":{"login":"brian-brazil","id":7115638,"avatar_url":"https://avatars.githubusercontent.com/u/7115638?v=3","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-10T17:04:15Z","updated_at":"2015-10-13T09:21:31Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/prometheus/prometheus/pulls/1161","html_url":"https://github.com/prometheus/prometheus/pull/1161","diff_url":"https://github.com/prometheus/prometheus/pull/1161.diff","patch_url":"https://github.com/prometheus/prometheus/pull/1161.patch"},"body":"This is an improvement from both a theoretical and practical\r\nstandpoint.\r\n\r\nTheory:\r\nFor simplicty I'll take the example of increase().\r\n\r\nCounters are fundamentally lossy, if there's a counter reset\r\nor instance failure between one scrape and the next we'll\r\nlose information about the period up to the reset/failure.\r\nThus the samples we have allow us to calculate a lower-bound\r\non the increase() in a time period.\r\n\r\nExtrapolation multiples this by an amount depending on timing which is\r\nan upper bound based on what might have happened if everything continued\r\nas it was.\r\nThis mix of upper and lower bounds means that we can't make any\r\nmeaningful statements about the output of increase() in relation to what\r\nactually happened.\r\n\r\nBy removing the extrapolation, we can once again reason that the result\r\nof increase() is a lower bound on the true increase of the counter.\r\n\r\nPractical:\r\nFixes #581.\r\nThe extrapolation presumes that it's looking at a range within a\r\ncontinuous stream of samples. If in fact the time series starts or\r\nend within this range, this leads to an over-correction.\r\n\r\nFor discrete counters and gauges, extrapolating to invalid values in\r\ntheir domain can be confusing and pervent rules being written that\r\ndepend on exact values.\r\n\r\nFor those looking to graph things more accurately irate() is a better\r\nchoice than extrapolation on rate().\r\nFor those looking to calculate how a gauge is trending deriv() is a\r\nbetter choice than delta().\r\n\r\n\r\n@juliusv @fabxc This depends on #1158"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/147659604","html_url":"https://github.com/prometheus/prometheus/pull/1161#issuecomment-147659604","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1161","id":147659604,"user":{"login":"brian-brazil","id":7115638,"avatar_url":"https://avatars.githubusercontent.com/u/7115638?v=3","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"created_at":"2015-10-13T09:21:31Z","updated_at":"2015-10-13T09:21:31Z","body":"> If we do this, we really cannot sell rate() as increase-per-second anymore.\r\n\r\nIf that's the case we can't sell what we currently have as increase per second either. What we currently have can be off by an factor up to between a scrape interval and the range.\r\n\r\nWithout extrapolation it can only be off by up to a scrape interval. This is a much smaller discrepancy, and is meaningful without having to look at the data.\r\n\r\n> As an example, if I scrape a counter every 15s and do a rate(foo[1m]) over it, I get a rate of 41/s without extrapolation, but 55/s with extrapolation, where the latter matches the true per-second counter increase pretty much perfectly and the unextrapolated result is way off. \r\n\r\nTaking your example, the current rate can produce 55/s when the true value is 13.75/s. This is not more accurate.\r\n\r\n> but will they understand that even at an averaging window 4x as big as their sampling interval, their rate will be off by 25% already?\r\n\r\nDo they understand the current behaviour can be off by between 25% and 300% in that case?\r\n\r\nAs with interpolation in evaluation this is something that sounds good in theory but doesn't work out in practice. We should never do any interpolation or extrapolation without informed opt-in by the user, as it can cause unexpected results and only the user knows if that's okay for their use case."}}