{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/67","labels_url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/67/comments","events_url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/67/events","html_url":"https://github.com/cloudfoundry/loggregator/issues/67","id":96257851,"number":67,"title":"TruncatingBuffer is unnecessarily fragile","user":{"login":"EngineerBetter","id":11817552,"avatar_url":"https://avatars.githubusercontent.com/u/11817552?v=3","gravatar_id":"","url":"https://api.github.com/users/EngineerBetter","html_url":"https://github.com/EngineerBetter","followers_url":"https://api.github.com/users/EngineerBetter/followers","following_url":"https://api.github.com/users/EngineerBetter/following{/other_user}","gists_url":"https://api.github.com/users/EngineerBetter/gists{/gist_id}","starred_url":"https://api.github.com/users/EngineerBetter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EngineerBetter/subscriptions","organizations_url":"https://api.github.com/users/EngineerBetter/orgs","repos_url":"https://api.github.com/users/EngineerBetter/repos","events_url":"https://api.github.com/users/EngineerBetter/events{/privacy}","received_events_url":"https://api.github.com/users/EngineerBetter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-21T08:24:37Z","updated_at":"2015-07-21T13:30:19Z","closed_at":null,"body":"`TruncatingBuffer` drops the whole buffer when full. This seems needlessly fragile, making a small change in workload (one extra message) have a far more destructive impact than it needs to.\r\n\r\nA less fragile implementation would discard the newest messages as they arrive. This approach would be better than discarding those at the front of the queue, as the older messages may have information that 'leads up to' the blockage.\r\n\r\nAn antifragile solution would dynamically resize the buffer when it approaches capacity, up to a larger and more sensible limit. "},"comment":{"url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/comments/123304202","html_url":"https://github.com/cloudfoundry/loggregator/issues/67#issuecomment-123304202","issue_url":"https://api.github.com/repos/cloudfoundry/loggregator/issues/67","id":123304202,"user":{"login":"ajackson","id":16051,"avatar_url":"https://avatars.githubusercontent.com/u/16051?v=3","gravatar_id":"","url":"https://api.github.com/users/ajackson","html_url":"https://github.com/ajackson","followers_url":"https://api.github.com/users/ajackson/followers","following_url":"https://api.github.com/users/ajackson/following{/other_user}","gists_url":"https://api.github.com/users/ajackson/gists{/gist_id}","starred_url":"https://api.github.com/users/ajackson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajackson/subscriptions","organizations_url":"https://api.github.com/users/ajackson/orgs","repos_url":"https://api.github.com/users/ajackson/repos","events_url":"https://api.github.com/users/ajackson/events{/privacy}","received_events_url":"https://api.github.com/users/ajackson/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T13:30:18Z","updated_at":"2015-07-21T13:30:18Z","body":"TruncatingBuffer is used as a \"circuit breaker\" to prevent server backup and slowdown when we have slow consumers.   \r\n\r\nYou are correct in that resizing the buffer would be a much more antifragile solution.   However it is so only in the case for a single client.   In the case of the server that has many (as in 1-10K+) of these buffers, resizing the buffer will place severe memory constraints on the server and make it's memory usage very unpredictable.   This will result in loss of service for not just one but many to all of the clients.   \r\n\r\nDiscarding the newest messages as they arrive is another great strategy.  In our environments we usually see either clients that keep up or clients that can't. This is usually by a large margin.   In the case where we only drop one of the messages, we don't see an ability for the client to catch backup and actually receive the messages.  We took the approach of dropping all the messages but the warning message as this gives the client a chance to catch back up with the incoming messages.\r\n"}}