{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/36","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/36/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/36/events","html_url":"https://github.com/logstash-plugins/logstash-input-lumberjack/issues/36","id":93583168,"number":36,"title":"logstash dies with OpenBreaker exception","user":{"login":"webmstr","id":4305995,"avatar_url":"https://avatars.githubusercontent.com/u/4305995?v=3","gravatar_id":"","url":"https://api.github.com/users/webmstr","html_url":"https://github.com/webmstr","followers_url":"https://api.github.com/users/webmstr/followers","following_url":"https://api.github.com/users/webmstr/following{/other_user}","gists_url":"https://api.github.com/users/webmstr/gists{/gist_id}","starred_url":"https://api.github.com/users/webmstr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/webmstr/subscriptions","organizations_url":"https://api.github.com/users/webmstr/orgs","repos_url":"https://api.github.com/users/webmstr/repos","events_url":"https://api.github.com/users/webmstr/events{/privacy}","received_events_url":"https://api.github.com/users/webmstr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-07-07T17:21:39Z","updated_at":"2015-10-09T05:30:45Z","closed_at":"2015-07-10T17:36:52Z","body":"About twice a day, logstash 1.5.2 will stop processing requests.  It logs this message:\r\n\r\n    {:timestamp=>\"2015-07-07T09:25:07.113000-0700\", :message=>\"Exception in lumberjack input thread\", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}\r\n\r\nThe only reference I see for this is [in the code](https://github.com/logstash-plugins/logstash-input-lumberjack/blob/master/lib/logstash/inputs/lumberjack.rb#L97), where it says, \"it's safe to ignore\".  Either that's not true, or not implemented well.\r\n\r\nThe comment also says that it's a result of too many errors.  No other errors are logged before this message appears.\r\n\r\nRestarting logstash by itself doesn't help much, because it doesn't seem to like waking up with 500 shippers trying to connect.  It will say \"Lumberjack input: the pipeline is blocked, temporary refusing new connection\" and throw more of these OpenBreaker errors (though not always fatally?)  Sometimes I have to shut down all the shippers and bring them back up slowly, allowing the backlog to clear before bringing more shippers online."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/comments/146758442","html_url":"https://github.com/logstash-plugins/logstash-input-lumberjack/issues/36#issuecomment-146758442","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-lumberjack/issues/36","id":146758442,"user":{"login":"fizerkhan","id":1433380,"avatar_url":"https://avatars.githubusercontent.com/u/1433380?v=3","gravatar_id":"","url":"https://api.github.com/users/fizerkhan","html_url":"https://github.com/fizerkhan","followers_url":"https://api.github.com/users/fizerkhan/followers","following_url":"https://api.github.com/users/fizerkhan/following{/other_user}","gists_url":"https://api.github.com/users/fizerkhan/gists{/gist_id}","starred_url":"https://api.github.com/users/fizerkhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fizerkhan/subscriptions","organizations_url":"https://api.github.com/users/fizerkhan/orgs","repos_url":"https://api.github.com/users/fizerkhan/repos","events_url":"https://api.github.com/users/fizerkhan/events{/privacy}","received_events_url":"https://api.github.com/users/fizerkhan/received_events","type":"User","site_admin":false},"created_at":"2015-10-09T05:30:45Z","updated_at":"2015-10-09T05:30:45Z","body":"+1"}}