{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/3897","labels_url":"https://api.github.com/repos/elastic/logstash/issues/3897/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/3897/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/3897/events","html_url":"https://github.com/elastic/logstash/issues/3897","id":105834555,"number":3897,"title":"Internal message bus","user":{"login":"ph","id":640,"avatar_url":"https://avatars.githubusercontent.com/u/640?v=3","gravatar_id":"","url":"https://api.github.com/users/ph","html_url":"https://github.com/ph","followers_url":"https://api.github.com/users/ph/followers","following_url":"https://api.github.com/users/ph/following{/other_user}","gists_url":"https://api.github.com/users/ph/gists{/gist_id}","starred_url":"https://api.github.com/users/ph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ph/subscriptions","organizations_url":"https://api.github.com/users/ph/orgs","repos_url":"https://api.github.com/users/ph/repos","events_url":"https://api.github.com/users/ph/events{/privacy}","received_events_url":"https://api.github.com/users/ph/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/design","name":"design","color":"eb6420"},{"url":"https://api.github.com/repos/elastic/logstash/labels/discuss","name":"discuss","color":"cc317c"},{"url":"https://api.github.com/repos/elastic/logstash/labels/metrics","name":"metrics","color":"006b75"},{"url":"https://api.github.com/repos/elastic/logstash/labels/v2.1.0","name":"v2.1.0","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-09-10T15:30:21Z","updated_at":"2015-09-14T13:34:47Z","closed_at":null,"body":"The agent, the pipeline and the plugins will emit some kind of metrics or performance stats.\r\nIf we generalize that action, we are actually emitting a message of the type `metric`.\r\n\r\nInstead of creating a custom collector for the metrics, I think we could create some kind of internal message bus that would take care of any internal messages that the system create and let consumer subscribes to theses messages and act upon them.\r\n\r\nThis would allow to decouple the message from that actual work. This could allow us to do some neat trick like dynamic resources allocation for slower plugins based on the metric data.\r\n\r\n**Really high level syntax**\r\n\r\n```ruby\r\nclass MetricMessage < Message\r\nend\r\n\r\nmetric = MetricMessage.new\r\n\r\nbus = MessageBus.new\r\nbus.send(metric)\r\n\r\n\r\nclass MetricInternalSnapshot < MessageBusConsumer\r\n  def on_message(message)\r\n  end\r\nend\r\n\r\nsnapshot = MetricInternalSnapshot.new() # create a local aggregated view for the web api?\r\nexternal = MetricExternalReporter.new() # Push metrics event to external system like elasticsearch or statsd\r\nadapt_pipeline = AdaptPipeline.new() # Listen to metrics events detect bottlenext and do resource reallocation\r\n\r\nbus.add_subscriber(snapshot, :type => MetricMessage)\r\nbus.add_subscriber(external, :type => MetricMessage)\r\nbus.add_subscribe(adapt_pipeline, :type => [MetricMessage, NeedResourceMessage])\r\n\r\n```\r\n\r\nThis architecture would also allow us to use the persistent queue and do replays of metrics if the system goes down and we need to.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/140077306","html_url":"https://github.com/elastic/logstash/issues/3897#issuecomment-140077306","issue_url":"https://api.github.com/repos/elastic/logstash/issues/3897","id":140077306,"user":{"login":"ph","id":640,"avatar_url":"https://avatars.githubusercontent.com/u/640?v=3","gravatar_id":"","url":"https://api.github.com/users/ph","html_url":"https://github.com/ph","followers_url":"https://api.github.com/users/ph/followers","following_url":"https://api.github.com/users/ph/following{/other_user}","gists_url":"https://api.github.com/users/ph/gists{/gist_id}","starred_url":"https://api.github.com/users/ph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ph/subscriptions","organizations_url":"https://api.github.com/users/ph/orgs","repos_url":"https://api.github.com/users/ph/repos","events_url":"https://api.github.com/users/ph/events{/privacy}","received_events_url":"https://api.github.com/users/ph/received_events","type":"User","site_admin":false},"created_at":"2015-09-14T13:34:47Z","updated_at":"2015-09-14T13:34:47Z","body":"@guyboertje What would be the avantages of using an array vs a class?"}}