{"action":"created","issue":{"url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/16","labels_url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/16/comments","events_url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/16/events","html_url":"https://github.com/remind101/logspout-kinesis/pull/16","id":104611142,"number":16,"title":"Fix pump timeout","user":{"login":"benguillet","id":766416,"avatar_url":"https://avatars.githubusercontent.com/u/766416?v=3","gravatar_id":"","url":"https://api.github.com/users/benguillet","html_url":"https://github.com/benguillet","followers_url":"https://api.github.com/users/benguillet/followers","following_url":"https://api.github.com/users/benguillet/following{/other_user}","gists_url":"https://api.github.com/users/benguillet/gists{/gist_id}","starred_url":"https://api.github.com/users/benguillet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benguillet/subscriptions","organizations_url":"https://api.github.com/users/benguillet/orgs","repos_url":"https://api.github.com/users/benguillet/repos","events_url":"https://api.github.com/users/benguillet/events{/privacy}","received_events_url":"https://api.github.com/users/benguillet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-03T02:55:14Z","updated_at":"2015-09-11T23:07:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/remind101/logspout-kinesis/pulls/16","html_url":"https://github.com/remind101/logspout-kinesis/pull/16","diff_url":"https://github.com/remind101/logspout-kinesis/pull/16.diff","patch_url":"https://github.com/remind101/logspout-kinesis/pull/16.patch"},"body":"The old architecture was blocking in some conditions and thus triggering a pump timeout in logspout, which results in logspout deleting the logstream channel and completely stoppping to send logs to the adapter.\r\n\r\nThis is a refactor using channels, select and better goroutines, so that it never blocks. If the flush takes too long and the flusher start to accumulate delay (more than 10 buffers in the queue), it starts dropping buffers (with a warning). This refactor also adds tests."},"comment":{"url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/comments/139683647","html_url":"https://github.com/remind101/logspout-kinesis/pull/16#issuecomment-139683647","issue_url":"https://api.github.com/repos/remind101/logspout-kinesis/issues/16","id":139683647,"user":{"login":"ejholmes","id":95166,"avatar_url":"https://avatars.githubusercontent.com/u/95166?v=3","gravatar_id":"","url":"https://api.github.com/users/ejholmes","html_url":"https://github.com/ejholmes","followers_url":"https://api.github.com/users/ejholmes/followers","following_url":"https://api.github.com/users/ejholmes/following{/other_user}","gists_url":"https://api.github.com/users/ejholmes/gists{/gist_id}","starred_url":"https://api.github.com/users/ejholmes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejholmes/subscriptions","organizations_url":"https://api.github.com/users/ejholmes/orgs","repos_url":"https://api.github.com/users/ejholmes/repos","events_url":"https://api.github.com/users/ejholmes/events{/privacy}","received_events_url":"https://api.github.com/users/ejholmes/received_events","type":"User","site_admin":false},"created_at":"2015-09-11T23:07:22Z","updated_at":"2015-09-11T23:07:22Z","body":"Aside from those comments, this looks pretty good to me.\r\n\r\nIt might make sense from a performance standpoint to eventually have a separate writer per partition key/container. Since we only need to guarantee ordering of log lines for an individual container, and the containers partition key would never change, we could spin up a separate writer/flusher in a go routine for each container. This means that goroutines would grow unbounded unless we could get events from logspout when a container dies though..."}}