{"action":"opened","issue":{"url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1179","labels_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1179/labels{/name}","comments_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1179/comments","events_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1179/events","html_url":"https://github.com/bosun-monitor/bosun/issues/1179","id":95992844,"number":1179,"title":"scollector: all wmi metrics stall on individual server for 3m every 16m ","user":{"login":"gbrayut","id":304401,"avatar_url":"https://avatars.githubusercontent.com/u/304401?v=3","gravatar_id":"","url":"https://api.github.com/users/gbrayut","html_url":"https://github.com/gbrayut","followers_url":"https://api.github.com/users/gbrayut/followers","following_url":"https://api.github.com/users/gbrayut/following{/other_user}","gists_url":"https://api.github.com/users/gbrayut/gists{/gist_id}","starred_url":"https://api.github.com/users/gbrayut/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbrayut/subscriptions","organizations_url":"https://api.github.com/users/gbrayut/orgs","repos_url":"https://api.github.com/users/gbrayut/repos","events_url":"https://api.github.com/users/gbrayut/events{/privacy}","received_events_url":"https://api.github.com/users/gbrayut/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"gbrayut","id":304401,"avatar_url":"https://avatars.githubusercontent.com/u/304401?v=3","gravatar_id":"","url":"https://api.github.com/users/gbrayut","html_url":"https://github.com/gbrayut","followers_url":"https://api.github.com/users/gbrayut/followers","following_url":"https://api.github.com/users/gbrayut/following{/other_user}","gists_url":"https://api.github.com/users/gbrayut/gists{/gist_id}","starred_url":"https://api.github.com/users/gbrayut/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbrayut/subscriptions","organizations_url":"https://api.github.com/users/gbrayut/orgs","repos_url":"https://api.github.com/users/gbrayut/repos","events_url":"https://api.github.com/users/gbrayut/events{/privacy}","received_events_url":"https://api.github.com/users/gbrayut/received_events","type":"User","site_admin":false},"milestone":null,"comments":0,"created_at":"2015-07-20T05:49:48Z","updated_at":"2015-07-20T05:49:48Z","closed_at":null,"body":"Just found a strange issue that was causing some of our alerts to go unknown. It appears that there is a stall in wmi mertrics on a few of our systems that lasts for about 3 minutes and occurs about every 16m. Here is a picture of the win.proc.cpu metric, which shows the gaps (some lasting longer than 3 minutes)\r\n\r\n![image](https://cloud.githubusercontent.com/assets/304401/8770484/f35aab9a-2e6f-11e5-8de5-52bce2084679.png)\r\n\r\nIt only seems to effect WMI metrics, as non-WMI metrics like scollector.collect.sent still work fine but show the count of sent metrics drops significantly during the stall:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/304401/8770505/34d4349c-2e70-11e5-9cf2-7786e14ddc92.png)\r\n\r\nI tried running an older version of scollector that was built back in May, and it also had the same issue. Even stranger is the issue occurred in both instances of scollector (at the exact same time!), even when the old version was pointed at our devbosun instance.\r\n\r\nI ran the second instance using the `-p -f c_cpu_windows` flags, and you can see the metrics were working fine (every 15s) and then just stopped win no error or reason.\r\n\r\n```\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370600,\"value\":344967,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"interrupt\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370600,\"value\":145890642,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"user\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370600,\"value\":7385101,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu.interrupts\",\"timestamp\":1437370600,\"value\":1115497120,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu.dpcs\",\"timestamp\":1437370600,\"value\":0,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"os.cpu\",\"timestamp\":1437370600,\"value\":185507239,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370600,\"value\":13106454,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"privileged\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370600,\"value\":435323,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"interrupt\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370600,\"value\":171965461,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"user\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370600,\"value\":112945881,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total.interrupts\",\"timestamp\":1437370600,\"value\":935683479,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu_total.dpcs\",\"timestamp\":1437370600,\"value\":23,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370600,\"value\":112945881,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c1\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370600,\"value\":0,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c2\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370600,\"value\":0,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c3\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"scollector.collector.duration\",\"timestamp\":1437370600,\"value\":0.058055800000000005,\"tags\":{\"collector\":\"bosun.org/cmd/scollector/collectors.c_cpu_windows\",\"host\"\r\n:\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:41 info: queue.go:65: {\"metric\":\"scollector.collector.error\",\"timestamp\":1437370600,\"value\":0,\"tags\":{\"collector\":\"bosun.org/cmd/scollector/collectors.c_cpu_windows\",\"host\":\"ny-logsql01\",\"os\":\"w\r\nindows\"}}\r\n2015/07/20 05:36:55 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370615,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:55 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:55 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370615,\"value\":20304,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:55 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370615,\"value\":2,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:55 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370615,\"value\":416496,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":18788023,\"tags\":{\"cpu\":\"0\",\"host\":\"ny-logsql01\",\"type\":\"privileged\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":935810,\"tags\":{\"cpu\":\"0\",\"host\":\"ny-logsql01\",\"type\":\"interrupt\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":175517060,\"tags\":{\"cpu\":\"0\",\"host\":\"ny-logsql01\",\"type\":\"user\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":15675023,\"tags\":{\"cpu\":\"0\",\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.interrupts\",\"timestamp\":1437370615,\"value\":2718855591,\"tags\":{\"cpu\":\"0\",\"host\":\"ny-logsql01\"}}\r\n...omitted...\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":23313027,\"tags\":{\"cpu\":\"38\",\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.interrupts\",\"timestamp\":1437370615,\"value\":1388187020,\"tags\":{\"cpu\":\"38\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.dpcs\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"cpu\":\"38\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":12411692,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"privileged\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":344968,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"interrupt\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":145891273,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"user\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu\",\"timestamp\":1437370615,\"value\":7385959,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.interrupts\",\"timestamp\":1437370615,\"value\":1115515540,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.dpcs\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"cpu\":\"39\",\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"os.cpu\",\"timestamp\":1437370615,\"value\":185507281,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370615,\"value\":13106459,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"privileged\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370615,\"value\":435323,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"interrupt\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370615,\"value\":171965498,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"user\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total\",\"timestamp\":1437370615,\"value\":112947335,\"tags\":{\"host\":\"ny-logsql01\",\"type\":\"idle\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total.interrupts\",\"timestamp\":1437370615,\"value\":936510034,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu_total.dpcs\",\"timestamp\":1437370615,\"value\":27,\"tags\":{\"host\":\"ny-logsql01\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370615,\"value\":112947335,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c1\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c2\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"win.cpu.time_cstate\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"cpu\":\"total\",\"host\":\"ny-logsql01\",\"type\":\"c3\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"scollector.collector.duration\",\"timestamp\":1437370615,\"value\":0.057055100000000004,\"tags\":{\"collector\":\"bosun.org/cmd/scollector/collectors.c_cpu_windows\",\"host\"\r\n:\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:36:56 info: queue.go:65: {\"metric\":\"scollector.collector.error\",\"timestamp\":1437370615,\"value\":0,\"tags\":{\"collector\":\"bosun.org/cmd/scollector/collectors.c_cpu_windows\",\"host\":\"ny-logsql01\",\"os\":\"w\r\nindows\"}}\r\n2015/07/20 05:37:10 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370630,\"value\":20561,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:10 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370630,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:10 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370630,\"value\":415904,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:10 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370630,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:10 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370630,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:25 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370645,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:25 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370645,\"value\":20566,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:25 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370645,\"value\":2,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:25 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370645,\"value\":428480,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:25 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370645,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:40 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370660,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:40 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370660,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:40 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370660,\"value\":20571,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:40 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370660,\"value\":2,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:40 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370660,\"value\":440768,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:55 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370675,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:55 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370675,\"value\":452704,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:55 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370675,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:55 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370675,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:37:55 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370675,\"value\":20576,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:10 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370690,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:10 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370690,\"value\":20581,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:10 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370690,\"value\":2,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:10 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370690,\"value\":465072,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:10 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370690,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:25 info: queue.go:65: {\"metric\":\"scollector.collect.sent\",\"timestamp\":1437370705,\"value\":20586,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:25 info: queue.go:65: {\"metric\":\"scollector.collect.queued\",\"timestamp\":1437370705,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:25 info: queue.go:65: {\"metric\":\"scollector.collect.alloc\",\"timestamp\":1437370705,\"value\":477152,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:25 info: queue.go:65: {\"metric\":\"scollector.collect.goroutines\",\"timestamp\":1437370705,\"value\":14,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n2015/07/20 05:38:25 info: queue.go:65: {\"metric\":\"scollector.collect.dropped\",\"timestamp\":1437370705,\"value\":0,\"tags\":{\"host\":\"ny-logsql01\",\"os\":\"windows\"}}\r\n```\r\n\r\nTomorrow I think I will add pprof to scollector using a address/port config item to enable it so we can see what is happening in the stack traces."}}