{"action":"opened","issue":{"url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/3","labels_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/3/comments","events_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/3/events","html_url":"https://github.com/raintank/graphite-kairosdb/issues/3","id":78731420,"number":3,"title":"newest point in series null","user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-05-20T22:47:23Z","updated_at":"2015-05-20T22:47:23Z","closed_at":null,"body":"this is incredible.  and the exact opposite of #2 \r\nin fact, i only found out #2 by trying to reproduce this one in a script.\r\n\r\nanyway, in the alerting branch, we given a certain number of seconds margin to let the data flow in.\r\nthis is currently configured to 30 seconds.\r\nso when creating alert check jobs, we create a job that queries a range that ends at least 30s ago, and starts x steps before the end.\r\n\r\nsee https://github.com/raintank/grafana/blob/litmus-alerting-v3/pkg/alerting/scheduler.go#L72-78 for more details.\r\n\r\nhowever I noticed that very often, the *last* point in the returned range is null.\r\nsometimes it isn't, and it's also irrespective of the safety margin. i've tried 40, 50, 60, 100, 120, all with the same result: the last value is very often null.\r\n\r\nhere's some proof/illustration:\r\n\r\nshortly after 18:34:15 I got these log messages from grafana:\r\n(look at the last line first, it describes the job just executed, and the first part describes the result that was returned by graphite)\r\n\r\n```\r\n&{2015-05-20 22:33:15 +0000 UTC 2015-05-20 22:33:45 +0000 UTC [sum(dieter_plaetinck_be.*.network.ping.error_state)] <nil>} GRAPHITE START\r\n(graphite.Response) (len=1 cap=4) {\r\n (graphite.Series) {\r\n  Datapoints: ([]graphite.DataPoint) (len=4 cap=4) {\r\n   (graphite.DataPoint) (len=2 cap=4) {\r\n    (json.Number) (len=1) 0,\r\n    (json.Number) (len=10) 1432161195\r\n   },\r\n   (graphite.DataPoint) (len=2 cap=4) {\r\n    (json.Number) (len=1) 0,\r\n    (json.Number) (len=10) 1432161205\r\n   },\r\n   (graphite.DataPoint) (len=2 cap=4) {\r\n    (json.Number) (len=1) 0,\r\n    (json.Number) (len=10) 1432161215\r\n   },\r\n   (graphite.DataPoint) (len=2 cap=4) {\r\n    (json.Number) ,\r\n    (json.Number) (len=10) 1432161225\r\n   }\r\n  },\r\n  Target: (string) (len=57) \"sumSeries(dieter_plaetinck_be.*.network.ping.error_state)\"\r\n }\r\n}\r\njob results <Job> key=alert-id_1432161255 generatedAt=2015-05-20 22:34:15.00006052 +0000 UTC lastPointTs=2015-05-20 22:33:45 +0000 UTC definition: <CheckDef> Crit: ''sum(graphite(\"sum(dieter_plaetinck_be.*.network.ping.error_state)\", \"30s\", \"\", \"\") >= 1) == 3' -- Warn: '0' GraphiteContext saw 1 unknown values returned from server Unknown\r\n```\r\n\r\nSo, we got this job, generated at `2015-05-20 22:34:15.00006052 +0000 UTC`\r\nIt queries for a range that ends at `2015-05-20 22:33:45 +0000 UTC` ,  (i.e. 30 seconds ago)\r\nAll points have a value, except for the last one\r\n```\r\n date -d @1432161225\r\nWed May 20 18:33:45 EDT 2015\r\n```\r\n\r\ni also used wireshark to make double sure, which basically confirms:\r\n\r\n```\r\nGET /render/?format=json&from=1432161195&target=sum%28dieter_plaetinck_be.%2A.network.ping.error_state%29&until=1432161225 HTTP/1.1\r\nHost: graphite-api:8888\r\nUser-Agent: Go 1.1 package http\r\nX-Org-Id: 1\r\nAccept-Encoding: gzip\r\n\r\nHTTP/1.1 200 OK\r\nServer: gunicorn/19.3.0\r\nDate: Wed, 20 May 2015 22:34:15 GMT\r\nConnection: close\r\nLast-Modified: Wed, 20 May 2015 22:34:15 GMT\r\nExpires: Wed, 20 May 2015 22:35:15 GMT\r\nContent-Type: application/json\r\nCache-Control: max-age=60\r\nContent-Length: 160\r\n\r\n[{\"target\": \"sumSeries(dieter_plaetinck_be.*.network.ping.error_state)\", \"datapoints\": [[0, 1432161195], [0, 1432161205], [0, 1432161215], [null, 1432161225]]}]\r\n```"}}