{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1114","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1114/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1114/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1114/events","html_url":"https://github.com/prometheus/prometheus/issues/1114","id":108210369,"number":1114,"title":"High CPU after config change and SIGHUP","user":{"login":"dan-cleinmark","id":3299293,"avatar_url":"https://avatars.githubusercontent.com/u/3299293?v=3","gravatar_id":"","url":"https://api.github.com/users/dan-cleinmark","html_url":"https://github.com/dan-cleinmark","followers_url":"https://api.github.com/users/dan-cleinmark/followers","following_url":"https://api.github.com/users/dan-cleinmark/following{/other_user}","gists_url":"https://api.github.com/users/dan-cleinmark/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-cleinmark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-cleinmark/subscriptions","organizations_url":"https://api.github.com/users/dan-cleinmark/orgs","repos_url":"https://api.github.com/users/dan-cleinmark/repos","events_url":"https://api.github.com/users/dan-cleinmark/events{/privacy}","received_events_url":"https://api.github.com/users/dan-cleinmark/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/prometheus/prometheus/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/prometheus/prometheus/labels/Critical","name":"Critical","color":"e102d8"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/prometheus/prometheus/milestones/1","html_url":"https://github.com/prometheus/prometheus/milestones/v0.16.0","labels_url":"https://api.github.com/repos/prometheus/prometheus/milestones/1/labels","id":1317081,"number":1,"title":"v0.16.0","description":"","creator":{"login":"fabxc","id":4948210,"avatar_url":"https://avatars.githubusercontent.com/u/4948210?v=3","gravatar_id":"","url":"https://api.github.com/users/fabxc","html_url":"https://github.com/fabxc","followers_url":"https://api.github.com/users/fabxc/followers","following_url":"https://api.github.com/users/fabxc/following{/other_user}","gists_url":"https://api.github.com/users/fabxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fabxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabxc/subscriptions","organizations_url":"https://api.github.com/users/fabxc/orgs","repos_url":"https://api.github.com/users/fabxc/repos","events_url":"https://api.github.com/users/fabxc/events{/privacy}","received_events_url":"https://api.github.com/users/fabxc/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":0,"state":"open","created_at":"2015-09-21T19:04:21Z","updated_at":"2015-09-24T21:53:17Z","due_on":"2015-09-29T22:00:00Z","closed_at":null},"comments":6,"created_at":"2015-09-24T20:42:55Z","updated_at":"2015-09-25T12:57:13Z","closed_at":null,"body":"We're seeing very high CPU use on all cores after a SIGHUP if the prometheus config file changes.  Originally we were on 0.15.1 rev. 7a6d12a and the issue still persists on 0.16.0rc1.  CPU performance is as expected when the service first starts but consistently drops off after a config change & SIGHUP. A screenshot showing CPU load after a config change & HUP is below.\r\n\r\nThe green line shows CPU cores idle.  The first drop shows an addition in the global labels followed by a HUP, the next 2 are HUPing prometheus after removing target groups - the increases correlate to restarts of the prometheus service.  When CPU spikes after a config reload, we see ~ 4/4 cores in 'user' state and on a 16 core box we see ~25% user, ~25% system and ~50% iowait - interestingly, disk load was always constantly stable, even though CPU iowait was very high.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/3299293/10085085/8cad376e-62cc-11e5-8c2c-09455e44c88e.png)\r\n\r\nOur prometheus config (200 total file_sd_config groups):\r\n\r\n    global:\r\n      evaluation_interval: 15s\r\n      labels:\r\n        env: qa1\r\n      scrape_interval: 15s\r\n      scrape_timeout: 5s\r\n    rule_files: [/etc/prometheus/prometheus.rules, /etc/prometheus/common.rules, /etc/prometheus/elastic-search.rules,\r\n      /etc/prometheus/test.rules, /etc/prometheus/third-party-services.rules, /etc/prometheus/jenkins.rules,\r\n      /etc/prometheus/team-abc.rules, /etc/prometheus/team-123.rules]\r\n    scrape_configs:\r\n    - job_name: services_health\r\n      target_groups:\r\n      - targets: ['localhost:8180']\r\n    - file_sd_configs:\r\n      - names: [/etc/prometheus/sd/service1.node.yml]\r\n      job_name: service1_node\r\n      metrics_path: /9100/metrics\r\n      scrape_interval: 5s\r\n    - file_sd_configs:\r\n      - names: [/etc/prometheus/sd/service1.app.yml]\r\n      job_name: service1\r\n      metrics_path: /9100/metrics\r\n      scrape_interval: 5s\r\n      - file_sd_configs:\r\n        - names: [/etc/prometheus/sd/service2.node.yml]\r\n        job_name: service1_node\r\n        metrics_path: /9100/metrics\r\n        scrape_interval: 5s\r\n      - file_sd_configs:\r\n        - names: [/etc/prometheus/sd/service2.app.yml]\r\n        job_name: service1\r\n        metrics_path: /9100/metrics\r\n        scrape_interval: 5s\r\n    ..."},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/143214412","html_url":"https://github.com/prometheus/prometheus/issues/1114#issuecomment-143214412","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1114","id":143214412,"user":{"login":"juliusv","id":538008,"avatar_url":"https://avatars.githubusercontent.com/u/538008?v=3","gravatar_id":"","url":"https://api.github.com/users/juliusv","html_url":"https://github.com/juliusv","followers_url":"https://api.github.com/users/juliusv/followers","following_url":"https://api.github.com/users/juliusv/following{/other_user}","gists_url":"https://api.github.com/users/juliusv/gists{/gist_id}","starred_url":"https://api.github.com/users/juliusv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juliusv/subscriptions","organizations_url":"https://api.github.com/users/juliusv/orgs","repos_url":"https://api.github.com/users/juliusv/repos","events_url":"https://api.github.com/users/juliusv/events{/privacy}","received_events_url":"https://api.github.com/users/juliusv/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T12:57:13Z","updated_at":"2015-09-25T12:57:13Z","body":"@fabxc Hmm, ok. But looking at https://github.com/prometheus/prometheus/pull/1064/files, you actually did change exactly this code to always return in case of a closed target group channel. I still have to understand why the `done` channel wouldn't always be closed at the same time though."}}