{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/2632","labels_url":"https://api.github.com/repos/elastic/logstash/issues/2632/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/2632/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/2632/events","html_url":"https://github.com/elastic/logstash/issues/2632","id":57987567,"number":2632,"title":"Add support for clustering Logstash instances","user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/enhancement","name":"enhancement","color":"009800"},{"url":"https://api.github.com/repos/elastic/logstash/labels/manageability","name":"manageability","color":"c7def8"},{"url":"https://api.github.com/repos/elastic/logstash/labels/roadmap","name":"roadmap","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/logstash/labels/v2.0.0","name":"v2.0.0","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-17T21:47:43Z","updated_at":"2015-08-04T22:40:38Z","closed_at":null,"body":"Today, each Logstash instance is a *full pipeline* -- inputs, filters and outputs stages. In large-scale Logstash deployments, users run multiple instances of Logstash in order to horizontally scale event processing. This requires manual management of individual configuration files, or custom/3rd party configuration automation tools such as Puppet or Chef. \r\n\r\nWe plan to introduce a concept of a *Logstash cluster*, where instances can be controlled as a whole (on a cluster level), instead of being separate parts. This would entail the following features:\r\n\r\n1. Provide an option to centrally store a Logstash config, which is shared across all the instances in the cluster. This would be the single source of truth for all the instances\r\n2. Provide APIs to control the cluster, dynamically, to change configuration. See #2612 \r\n3. Provide APIs to monitor instances at the cluster level. See #2611 \r\n \r\nLogstash can still be started in a single-instance, non-clustered mode; file based configuration will continue to work. \r\n\r\nClustering instances will also provide the necessary groundwork for potential long-term enhancements like automatic load balancing, failover, running multiple pipelines and so on. "},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/127781765","html_url":"https://github.com/elastic/logstash/issues/2632#issuecomment-127781765","issue_url":"https://api.github.com/repos/elastic/logstash/issues/2632","id":127781765,"user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T22:40:38Z","updated_at":"2015-08-04T22:40:38Z","body":"@bitsofinfo thats exactly our thinking...we'll make it pluggable, so its easy to add in an alternate implementation for a config store. The first implementation will use ES as a config store."}}