{"action":"created","issue":{"url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/728","labels_url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/728/labels{/name}","comments_url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/728/comments","events_url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/728/events","html_url":"https://github.com/bloomberg/chef-bcpc/pull/728","id":96671759,"number":728,"title":"Organize Ceph and RGW Zabbix checks","user":{"login":"kelvk","id":9847174,"avatar_url":"https://avatars.githubusercontent.com/u/9847174?v=3","gravatar_id":"","url":"https://api.github.com/users/kelvk","html_url":"https://github.com/kelvk","followers_url":"https://api.github.com/users/kelvk/followers","following_url":"https://api.github.com/users/kelvk/following{/other_user}","gists_url":"https://api.github.com/users/kelvk/gists{/gist_id}","starred_url":"https://api.github.com/users/kelvk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kelvk/subscriptions","organizations_url":"https://api.github.com/users/kelvk/orgs","repos_url":"https://api.github.com/users/kelvk/repos","events_url":"https://api.github.com/users/kelvk/events{/privacy}","received_events_url":"https://api.github.com/users/kelvk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"kamidzi","id":8375625,"avatar_url":"https://avatars.githubusercontent.com/u/8375625?v=3","gravatar_id":"","url":"https://api.github.com/users/kamidzi","html_url":"https://github.com/kamidzi","followers_url":"https://api.github.com/users/kamidzi/followers","following_url":"https://api.github.com/users/kamidzi/following{/other_user}","gists_url":"https://api.github.com/users/kamidzi/gists{/gist_id}","starred_url":"https://api.github.com/users/kamidzi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kamidzi/subscriptions","organizations_url":"https://api.github.com/users/kamidzi/orgs","repos_url":"https://api.github.com/users/kamidzi/repos","events_url":"https://api.github.com/users/kamidzi/events{/privacy}","received_events_url":"https://api.github.com/users/kamidzi/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-07-22T21:40:53Z","updated_at":"2015-07-29T16:01:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bloomberg/chef-bcpc/pulls/728","html_url":"https://github.com/bloomberg/chef-bcpc/pull/728","diff_url":"https://github.com/bloomberg/chef-bcpc/pull/728.diff","patch_url":"https://github.com/bloomberg/chef-bcpc/pull/728.patch"},"body":"The following changes are introduced with this PR:\r\n1. Increase RGW logging so HTTP return codes are appropriately logged. This will be in place until http://tracker.ceph.com/issues/12432 is resolved.\r\n2. Monitor Civetweb logs if it's the RGW frontend in use.\r\n3. Monitor `ceph health` on head nodes only.\r\n4. Remove unnecessary Zabbix items (i.e. HTTP metrics) as these are better suited to be counted and presented in Kibana/Graphite.\r\n\r\nIn Zabbix [2.2.6](http://www.zabbix.com/rn2.2.6.php), `logtimefmt` field is added to all exported items. Since `zabbix_bcpc_templates.xml` was generated while we were on 2.2.2, the field addition is expected and can be safely ignored.\r\n\r\nFor the updated Zabbix templates to take effect on a running cluster, one needs to drop `BCPC-Headnode` and `BCPC-Worknode` templates (ensuring `delete selected with linked elements` is checked), then re-associate hosts with the appropriate templates. This is clearly not ideal. I am increasingly tempted to upgrade Zabbix to [2.4](https://www.zabbix.com/documentation/2.4/manual/introduction/whatsnew240#option_to_remove_missing_resources_when_importing) to address this shortcoming. Something for another PR."},"comment":{"url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/comments/125999581","html_url":"https://github.com/bloomberg/chef-bcpc/pull/728#issuecomment-125999581","issue_url":"https://api.github.com/repos/bloomberg/chef-bcpc/issues/728","id":125999581,"user":{"login":"kelvk","id":9847174,"avatar_url":"https://avatars.githubusercontent.com/u/9847174?v=3","gravatar_id":"","url":"https://api.github.com/users/kelvk","html_url":"https://github.com/kelvk","followers_url":"https://api.github.com/users/kelvk/followers","following_url":"https://api.github.com/users/kelvk/following{/other_user}","gists_url":"https://api.github.com/users/kelvk/gists{/gist_id}","starred_url":"https://api.github.com/users/kelvk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kelvk/subscriptions","organizations_url":"https://api.github.com/users/kelvk/orgs","repos_url":"https://api.github.com/users/kelvk/repos","events_url":"https://api.github.com/users/kelvk/events{/privacy}","received_events_url":"https://api.github.com/users/kelvk/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T16:01:55Z","updated_at":"2015-07-29T16:01:55Z","body":"Expanding on test detail as suggested by @kamidzi:\r\n\r\n1. Most of the testing/verification is by eyeballing https://10.0.100.6/zabbix/latest.php. `ceph health` should only be associated with the head nodes.\r\n2. A bunch of RGW \"Bucket*\" and RGW log items no longer exist for all nodes (exact items are in the diff)\r\n3. `/etc/zabbix/zabbix_agentd.d/zabbix-rgw.conf` now exists on all RGW nodes (head+work). Log checks are added if Ceph frontend is Civetweb, which is the BCPC default. Monitoring Apache RGW logs is excluded as it is expected to be deprecated per #626.\r\n4. RGW will log HTTP status codes in `/var/log/ceph/radosgw.log` like so, due to above-mentioned Ceph bug:\r\n<pre>\r\n2015-07-29 15:06:06.815257 7f22367fc700  1 ====== req done req=0x39402c0 http_status=200 ======\r\n</pre>\r\n5. This PR cleans up various Ceph/RGW errors in `/etc/zabbix/zabbix_agentd.log`. If #720 is merged as well, the log should be totally clean."}}