{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/4074","labels_url":"https://api.github.com/repos/elastic/logstash/issues/4074/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/4074/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/4074/events","html_url":"https://github.com/elastic/logstash/issues/4074","id":112667780,"number":4074,"title":"[meta] codec stream identity","user":{"login":"colinsurprenant","id":2010,"avatar_url":"https://avatars.githubusercontent.com/u/2010?v=3","gravatar_id":"","url":"https://api.github.com/users/colinsurprenant","html_url":"https://github.com/colinsurprenant","followers_url":"https://api.github.com/users/colinsurprenant/followers","following_url":"https://api.github.com/users/colinsurprenant/following{/other_user}","gists_url":"https://api.github.com/users/colinsurprenant/gists{/gist_id}","starred_url":"https://api.github.com/users/colinsurprenant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinsurprenant/subscriptions","organizations_url":"https://api.github.com/users/colinsurprenant/orgs","repos_url":"https://api.github.com/users/colinsurprenant/repos","events_url":"https://api.github.com/users/colinsurprenant/events{/privacy}","received_events_url":"https://api.github.com/users/colinsurprenant/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/discuss","name":"discuss","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"guyboertje","id":84912,"avatar_url":"https://avatars.githubusercontent.com/u/84912?v=3","gravatar_id":"","url":"https://api.github.com/users/guyboertje","html_url":"https://github.com/guyboertje","followers_url":"https://api.github.com/users/guyboertje/followers","following_url":"https://api.github.com/users/guyboertje/following{/other_user}","gists_url":"https://api.github.com/users/guyboertje/gists{/gist_id}","starred_url":"https://api.github.com/users/guyboertje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyboertje/subscriptions","organizations_url":"https://api.github.com/users/guyboertje/orgs","repos_url":"https://api.github.com/users/guyboertje/repos","events_url":"https://api.github.com/users/guyboertje/events{/privacy}","received_events_url":"https://api.github.com/users/guyboertje/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-10-21T20:00:57Z","updated_at":"2015-10-22T16:21:12Z","closed_at":null,"body":"this issue follows up the discussion in logstash-plugins/logstash-codec-multiline#10 about supporting stream identity in codecs to solve the problem of using the multiline codec in conjunction with the file or lumberjack inputs for example.\r\n\r\nSince this issue involves many plugins and is an important issue to solve we should keep track of it here.\r\n\r\nMy [last suggestion](https://github.com/logstash-plugins/logstash-codec-multiline/issues/10#issuecomment-149993542) is to do the following:\r\n\r\n- replace the codec instantiation in the config to instead instantiate a new \"codec multiplexer\" class, which will ultimately expose a **backward compatible** decode method like `decode(data, stream_identity = nil)`\r\n- this new multiplexer `decode` method would take care of doing the bookkeeping of the stream identities vs actual codec instances.\r\n\r\nThis would allow to not touch/update all plugins where dealing with stream identities in codecs is not important and simply update the ones like file & lumberjack inputs to pass the values which makes sense as `stream_identity` to the `decode` method. for example for file input, the current code:\r\n```ruby\r\n    @tail.subscribe do |path, line|\r\n      @codec.decode(line) do |event|\r\n```\r\nwould become:\r\n```ruby\r\n    @tail.subscribe do |path, line|\r\n      @codec.decode(line, path) do |event|\r\n```\r\n\r\nFor the codecs nothing would change and we can all assume a single codec instance will always deal with a single consistent stream.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/150278700","html_url":"https://github.com/elastic/logstash/issues/4074#issuecomment-150278700","issue_url":"https://api.github.com/repos/elastic/logstash/issues/4074","id":150278700,"user":{"login":"ph","id":640,"avatar_url":"https://avatars.githubusercontent.com/u/640?v=3","gravatar_id":"","url":"https://api.github.com/users/ph","html_url":"https://github.com/ph","followers_url":"https://api.github.com/users/ph/followers","following_url":"https://api.github.com/users/ph/following{/other_user}","gists_url":"https://api.github.com/users/ph/gists{/gist_id}","starred_url":"https://api.github.com/users/ph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ph/subscriptions","organizations_url":"https://api.github.com/users/ph/orgs","repos_url":"https://api.github.com/users/ph/repos","events_url":"https://api.github.com/users/ph/events{/privacy}","received_events_url":"https://api.github.com/users/ph/received_events","type":"User","site_admin":false},"created_at":"2015-10-22T16:21:12Z","updated_at":"2015-10-22T16:21:12Z","body":"Lets talk about `stream_identity` life and their relation to the UDP/TCP/Lumberjack usage.\r\n\r\nRight now in the lumberjack input we clone the codec on each new connection, which would mean that the `stream_identity` would be valid until the connection dies. This can happen when we have back pressure. Do we want to make the `stream_identify` more resilient to theses can of scenario?\r\n\r\nIf we want to, the `stream_identity` would need to keep values with an expiration and do some book keeping?\r\n"}}