{"action":"reopened","number":89,"pull_request":{"url":"https://api.github.com/repos/heroku/logplex/pulls/89","id":14075007,"html_url":"https://github.com/heroku/logplex/pull/89","diff_url":"https://github.com/heroku/logplex/pull/89.diff","patch_url":"https://github.com/heroku/logplex/pull/89.patch","issue_url":"https://api.github.com/repos/heroku/logplex/issues/89","number":89,"state":"open","locked":false,"title":"Add localized channel ring buffer monitoring","user":{"login":"voidlock","id":8017,"avatar_url":"https://avatars.githubusercontent.com/u/8017?v=3","gravatar_id":"","url":"https://api.github.com/users/voidlock","html_url":"https://github.com/voidlock","followers_url":"https://api.github.com/users/voidlock/followers","following_url":"https://api.github.com/users/voidlock/following{/other_user}","gists_url":"https://api.github.com/users/voidlock/gists{/gist_id}","starred_url":"https://api.github.com/users/voidlock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voidlock/subscriptions","organizations_url":"https://api.github.com/users/voidlock/orgs","repos_url":"https://api.github.com/users/voidlock/repos","events_url":"https://api.github.com/users/voidlock/events{/privacy}","received_events_url":"https://api.github.com/users/voidlock/received_events","type":"User","site_admin":false},"body":"One of the main sources of load for Logplex is getting log messages into\r\nthe Redis ring buffer in order for some backlog to be visible when a\r\ncustomer requests their log tail. For apps which produce a reasonable\r\namount of data this works fine, but for apps which produce an excessive\r\namount of log data, it is both unnecessary and time-consuming to store\r\nthese log messages in Redis, because there's no need for a backlog when\r\nsimply watching the tail for a few seconds will give you plenty of data.\r\n\r\nThis commit adds node localized monitoring of the messages being sent to\r\nthe Redis ring buffer. A new flag `no_redis_local` has been to control\r\nthe automated channel monitoring. Localized monitoring has been added in\r\nsuch a way that it does not intefere with existing global channel flags.\r\nIn fact setting either `no_tail` or `no_redis` will take precedence and\r\nremove any localized flags that existed on the channel.\r\n\r\nThere are two application environment variables that can be set to\r\ncontrol the behaviour of this monitoring.\r\n\r\n- `channel_flood_msg_threshold` The messages per second required to\r\ntrigger flagging a channel as being flooded.\r\n- `channel_flood_flap_threshold` The number of consecutive flooding\r\nnotices required before actually setting the `no_redis_local` flag on\r\nthe channel.\r\n\r\nThe latter environment variable helps control flapping situations and\r\nhas been inplemented in such a way that if it requires 3 consecutive\r\nflood notices before taking action, then the system must also receive 3\r\nconsecutive notices of normal flow before disabling the `no_redis_local`\r\nflag.\r\n\r\nA couple tooling utilities were added to help with running the test\r\nsuites. These are only slightly modified versions of some tools Phil was\r\nalready using so credits for the bin scripts should go to him.\r\n\r\nAdditionally there are a few outstanding items that should be taken\r\ncare of or added. Those are:\r\n\r\n- The `logplex_worker` and `logplex_message` modules were modified but\r\nnot directly tested.\r\n- An additional stat record should be added to record the number of\r\ntimes the msg threshold is crossed and the number of times the actual\r\n`no_redis_local` flag is set on a channel. Some interesting stats could\r\nbe gathered to see how the automated system is performing and how it\r\ncould be tuned.","created_at":"2014-03-27T18:40:37Z","updated_at":"2015-05-12T18:21:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/heroku/logplex/pulls/89/commits","review_comments_url":"https://api.github.com/repos/heroku/logplex/pulls/89/comments","review_comment_url":"https://api.github.com/repos/heroku/logplex/pulls/comments{/number}","comments_url":"https://api.github.com/repos/heroku/logplex/issues/89/comments","statuses_url":"https://api.github.com/repos/heroku/logplex/statuses/fb8959ba792712949b611f56a04cb5def12e3c5e","head":{"label":"voidlock:big-brother","ref":"big-brother","sha":"fb8959ba792712949b611f56a04cb5def12e3c5e","user":{"login":"voidlock","id":8017,"avatar_url":"https://avatars.githubusercontent.com/u/8017?v=3","gravatar_id":"","url":"https://api.github.com/users/voidlock","html_url":"https://github.com/voidlock","followers_url":"https://api.github.com/users/voidlock/followers","following_url":"https://api.github.com/users/voidlock/following{/other_user}","gists_url":"https://api.github.com/users/voidlock/gists{/gist_id}","starred_url":"https://api.github.com/users/voidlock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voidlock/subscriptions","organizations_url":"https://api.github.com/users/voidlock/orgs","repos_url":"https://api.github.com/users/voidlock/repos","events_url":"https://api.github.com/users/voidlock/events{/privacy}","received_events_url":"https://api.github.com/users/voidlock/received_events","type":"User","site_admin":false},"repo":{"id":18121700,"name":"logplex","full_name":"voidlock/logplex","owner":{"login":"voidlock","id":8017,"avatar_url":"https://avatars.githubusercontent.com/u/8017?v=3","gravatar_id":"","url":"https://api.github.com/users/voidlock","html_url":"https://github.com/voidlock","followers_url":"https://api.github.com/users/voidlock/followers","following_url":"https://api.github.com/users/voidlock/following{/other_user}","gists_url":"https://api.github.com/users/voidlock/gists{/gist_id}","starred_url":"https://api.github.com/users/voidlock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voidlock/subscriptions","organizations_url":"https://api.github.com/users/voidlock/orgs","repos_url":"https://api.github.com/users/voidlock/repos","events_url":"https://api.github.com/users/voidlock/events{/privacy}","received_events_url":"https://api.github.com/users/voidlock/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/voidlock/logplex","description":"Heroku log router","fork":true,"url":"https://api.github.com/repos/voidlock/logplex","forks_url":"https://api.github.com/repos/voidlock/logplex/forks","keys_url":"https://api.github.com/repos/voidlock/logplex/keys{/key_id}","collaborators_url":"https://api.github.com/repos/voidlock/logplex/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/voidlock/logplex/teams","hooks_url":"https://api.github.com/repos/voidlock/logplex/hooks","issue_events_url":"https://api.github.com/repos/voidlock/logplex/issues/events{/number}","events_url":"https://api.github.com/repos/voidlock/logplex/events","assignees_url":"https://api.github.com/repos/voidlock/logplex/assignees{/user}","branches_url":"https://api.github.com/repos/voidlock/logplex/branches{/branch}","tags_url":"https://api.github.com/repos/voidlock/logplex/tags","blobs_url":"https://api.github.com/repos/voidlock/logplex/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/voidlock/logplex/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/voidlock/logplex/git/refs{/sha}","trees_url":"https://api.github.com/repos/voidlock/logplex/git/trees{/sha}","statuses_url":"https://api.github.com/repos/voidlock/logplex/statuses/{sha}","languages_url":"https://api.github.com/repos/voidlock/logplex/languages","stargazers_url":"https://api.github.com/repos/voidlock/logplex/stargazers","contributors_url":"https://api.github.com/repos/voidlock/logplex/contributors","subscribers_url":"https://api.github.com/repos/voidlock/logplex/subscribers","subscription_url":"https://api.github.com/repos/voidlock/logplex/subscription","commits_url":"https://api.github.com/repos/voidlock/logplex/commits{/sha}","git_commits_url":"https://api.github.com/repos/voidlock/logplex/git/commits{/sha}","comments_url":"https://api.github.com/repos/voidlock/logplex/comments{/number}","issue_comment_url":"https://api.github.com/repos/voidlock/logplex/issues/comments{/number}","contents_url":"https://api.github.com/repos/voidlock/logplex/contents/{+path}","compare_url":"https://api.github.com/repos/voidlock/logplex/compare/{base}...{head}","merges_url":"https://api.github.com/repos/voidlock/logplex/merges","archive_url":"https://api.github.com/repos/voidlock/logplex/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/voidlock/logplex/downloads","issues_url":"https://api.github.com/repos/voidlock/logplex/issues{/number}","pulls_url":"https://api.github.com/repos/voidlock/logplex/pulls{/number}","milestones_url":"https://api.github.com/repos/voidlock/logplex/milestones{/number}","notifications_url":"https://api.github.com/repos/voidlock/logplex/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/voidlock/logplex/labels{/name}","releases_url":"https://api.github.com/repos/voidlock/logplex/releases{/id}","created_at":"2014-03-26T00:31:13Z","updated_at":"2015-05-08T17:59:28Z","pushed_at":"2015-05-12T18:21:40Z","git_url":"git://github.com/voidlock/logplex.git","ssh_url":"git@github.com:voidlock/logplex.git","clone_url":"https://github.com/voidlock/logplex.git","svn_url":"https://github.com/voidlock/logplex","homepage":"","size":4615,"stargazers_count":0,"watchers_count":0,"language":"Erlang","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"heroku:master","ref":"master","sha":"4155557130c925b060fa99e1d047d32634d8bc7a","user":{"login":"heroku","id":23211,"avatar_url":"https://avatars.githubusercontent.com/u/23211?v=3","gravatar_id":"","url":"https://api.github.com/users/heroku","html_url":"https://github.com/heroku","followers_url":"https://api.github.com/users/heroku/followers","following_url":"https://api.github.com/users/heroku/following{/other_user}","gists_url":"https://api.github.com/users/heroku/gists{/gist_id}","starred_url":"https://api.github.com/users/heroku/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heroku/subscriptions","organizations_url":"https://api.github.com/users/heroku/orgs","repos_url":"https://api.github.com/users/heroku/repos","events_url":"https://api.github.com/users/heroku/events{/privacy}","received_events_url":"https://api.github.com/users/heroku/received_events","type":"Organization","site_admin":false},"repo":{"id":1165646,"name":"logplex","full_name":"heroku/logplex","owner":{"login":"heroku","id":23211,"avatar_url":"https://avatars.githubusercontent.com/u/23211?v=3","gravatar_id":"","url":"https://api.github.com/users/heroku","html_url":"https://github.com/heroku","followers_url":"https://api.github.com/users/heroku/followers","following_url":"https://api.github.com/users/heroku/following{/other_user}","gists_url":"https://api.github.com/users/heroku/gists{/gist_id}","starred_url":"https://api.github.com/users/heroku/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heroku/subscriptions","organizations_url":"https://api.github.com/users/heroku/orgs","repos_url":"https://api.github.com/users/heroku/repos","events_url":"https://api.github.com/users/heroku/events{/privacy}","received_events_url":"https://api.github.com/users/heroku/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/heroku/logplex","description":"Heroku log router","fork":false,"url":"https://api.github.com/repos/heroku/logplex","forks_url":"https://api.github.com/repos/heroku/logplex/forks","keys_url":"https://api.github.com/repos/heroku/logplex/keys{/key_id}","collaborators_url":"https://api.github.com/repos/heroku/logplex/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/heroku/logplex/teams","hooks_url":"https://api.github.com/repos/heroku/logplex/hooks","issue_events_url":"https://api.github.com/repos/heroku/logplex/issues/events{/number}","events_url":"https://api.github.com/repos/heroku/logplex/events","assignees_url":"https://api.github.com/repos/heroku/logplex/assignees{/user}","branches_url":"https://api.github.com/repos/heroku/logplex/branches{/branch}","tags_url":"https://api.github.com/repos/heroku/logplex/tags","blobs_url":"https://api.github.com/repos/heroku/logplex/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/heroku/logplex/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/heroku/logplex/git/refs{/sha}","trees_url":"https://api.github.com/repos/heroku/logplex/git/trees{/sha}","statuses_url":"https://api.github.com/repos/heroku/logplex/statuses/{sha}","languages_url":"https://api.github.com/repos/heroku/logplex/languages","stargazers_url":"https://api.github.com/repos/heroku/logplex/stargazers","contributors_url":"https://api.github.com/repos/heroku/logplex/contributors","subscribers_url":"https://api.github.com/repos/heroku/logplex/subscribers","subscription_url":"https://api.github.com/repos/heroku/logplex/subscription","commits_url":"https://api.github.com/repos/heroku/logplex/commits{/sha}","git_commits_url":"https://api.github.com/repos/heroku/logplex/git/commits{/sha}","comments_url":"https://api.github.com/repos/heroku/logplex/comments{/number}","issue_comment_url":"https://api.github.com/repos/heroku/logplex/issues/comments{/number}","contents_url":"https://api.github.com/repos/heroku/logplex/contents/{+path}","compare_url":"https://api.github.com/repos/heroku/logplex/compare/{base}...{head}","merges_url":"https://api.github.com/repos/heroku/logplex/merges","archive_url":"https://api.github.com/repos/heroku/logplex/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/heroku/logplex/downloads","issues_url":"https://api.github.com/repos/heroku/logplex/issues{/number}","pulls_url":"https://api.github.com/repos/heroku/logplex/pulls{/number}","milestones_url":"https://api.github.com/repos/heroku/logplex/milestones{/number}","notifications_url":"https://api.github.com/repos/heroku/logplex/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/heroku/logplex/labels{/name}","releases_url":"https://api.github.com/repos/heroku/logplex/releases{/id}","created_at":"2010-12-13T19:19:10Z","updated_at":"2015-05-08T18:34:24Z","pushed_at":"2015-05-11T23:29:51Z","git_url":"git://github.com/heroku/logplex.git","ssh_url":"git@github.com:heroku/logplex.git","clone_url":"https://github.com/heroku/logplex.git","svn_url":"https://github.com/heroku/logplex","homepage":"","size":7250,"stargazers_count":573,"watchers_count":573,"language":"Erlang","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":43,"mirror_url":null,"open_issues_count":22,"forks":43,"open_issues":22,"watchers":573,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/heroku/logplex/pulls/89"},"html":{"href":"https://github.com/heroku/logplex/pull/89"},"issue":{"href":"https://api.github.com/repos/heroku/logplex/issues/89"},"comments":{"href":"https://api.github.com/repos/heroku/logplex/issues/89/comments"},"review_comments":{"href":"https://api.github.com/repos/heroku/logplex/pulls/89/comments"},"review_comment":{"href":"https://api.github.com/repos/heroku/logplex/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/heroku/logplex/pulls/89/commits"},"statuses":{"href":"https://api.github.com/repos/heroku/logplex/statuses/fb8959ba792712949b611f56a04cb5def12e3c5e"}},"merged":false,"mergeable":false,"mergeable_state":"dirty","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":235,"deletions":23,"changed_files":9}}