{"action":"created","issue":{"url":"https://api.github.com/repos/chef/chef-server/issues/570","labels_url":"https://api.github.com/repos/chef/chef-server/issues/570/labels{/name}","comments_url":"https://api.github.com/repos/chef/chef-server/issues/570/comments","events_url":"https://api.github.com/repos/chef/chef-server/issues/570/events","html_url":"https://github.com/chef/chef-server/pull/570","id":111490328,"number":570,"title":"rabbitmq queue monitor + queue stability enhancements","user":{"login":"metadave","id":58244,"avatar_url":"https://avatars.githubusercontent.com/u/58244?v=3","gravatar_id":"","url":"https://api.github.com/users/metadave","html_url":"https://github.com/metadave","followers_url":"https://api.github.com/users/metadave/followers","following_url":"https://api.github.com/users/metadave/following{/other_user}","gists_url":"https://api.github.com/users/metadave/gists{/gist_id}","starred_url":"https://api.github.com/users/metadave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/metadave/subscriptions","organizations_url":"https://api.github.com/users/metadave/orgs","repos_url":"https://api.github.com/users/metadave/repos","events_url":"https://api.github.com/users/metadave/events{/privacy}","received_events_url":"https://api.github.com/users/metadave/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/chef/chef-server/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/chef/chef-server/labels/Signed%20CLA","name":"Signed CLA","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-14T20:50:19Z","updated_at":"2015-10-16T20:58:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/chef/chef-server/pulls/570","html_url":"https://github.com/chef/chef-server/pull/570","diff_url":"https://github.com/chef/chef-server/pull/570.diff","patch_url":"https://github.com/chef/chef-server/pull/570.patch"},"body":"This PR adds several enhancements for RabbitMQ stability.\r\n\r\n1. a `max-length` policy is set on the `/analytics` exchange\r\n2. The RabbitMQ Management console is enabled by default with SSL on port `15672`. A `rabbitmgmt` user is created to access this console.\r\n3. A Queue Length monitoring gen\\_server (`chef_wm_actions_queue_monitoring.erl`). This gen\\_server periodically connects to the management endpoint to check max-length and current length of the analytics queue. If the queue is at > 80% capacity, a warning is issued via lager. If the queue reaches 100% capacity, toggle a \"queue at capacity\" flag until messages have been read/purged from the queue. Code that publishes messages to RabbitMQ can check the state of the \"queue at capacity\" flag before enqueuing. \r\n4. Queue length statistics are reporting via the `_status` endpoint.\r\n\r\nMore details are available in `chef_wm_actions_queue_monitoring.erl`.\r\n\r\n---\r\n### Notes\r\n\r\n* This PR adds the PropER testing tool, which is ***GPL V3***. HOWEVER, this is a ***test-only dependency***.\r\n\r\n* SSL for the RabbitMQ Management has been configured to only accept `tlsv1.2` and `tlsv1.1` (POODLE, BEAST) via:\r\n\r\n\r\n\t\t{ssl, [{versions, ['tlsv1.2', 'tlsv1.1']}]}\r\n\r\n---\r\n\r\n### TODO: \r\n\r\n- [x] tests in `chef_wm_status_tests.erl`\r\n- [ ] integration tests/local rabbitmq setup\r\n\r\n\t"},"comment":{"url":"https://api.github.com/repos/chef/chef-server/issues/comments/148835491","html_url":"https://github.com/chef/chef-server/pull/570#issuecomment-148835491","issue_url":"https://api.github.com/repos/chef/chef-server/issues/570","id":148835491,"user":{"login":"marcparadise","id":1130204,"avatar_url":"https://avatars.githubusercontent.com/u/1130204?v=3","gravatar_id":"","url":"https://api.github.com/users/marcparadise","html_url":"https://github.com/marcparadise","followers_url":"https://api.github.com/users/marcparadise/followers","following_url":"https://api.github.com/users/marcparadise/following{/other_user}","gists_url":"https://api.github.com/users/marcparadise/gists{/gist_id}","starred_url":"https://api.github.com/users/marcparadise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcparadise/subscriptions","organizations_url":"https://api.github.com/users/marcparadise/orgs","repos_url":"https://api.github.com/users/marcparadise/repos","events_url":"https://api.github.com/users/marcparadise/events{/privacy}","received_events_url":"https://api.github.com/users/marcparadise/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T20:58:27Z","updated_at":"2015-10-16T20:58:27Z","body":"This looks good, with some relatively minor comments above. \r\n\r\nMy only large area of concern is the volume of requests this will be handling.  The timeout option made me think of it, but realistically if we ever hit that timeout we're already going to be in a really bad state.   \r\n\r\nThis is very lightweight as implemented, and I'm probably being overly paranoid - I'd feel better if we had decent integrated load suite that didn't require spinning up a ton of AWS nodes. \r\n\r\n"}}