{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/3416","labels_url":"https://api.github.com/repos/elastic/logstash/issues/3416/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/3416/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/3416/events","html_url":"https://github.com/elastic/logstash/issues/3416","id":87133105,"number":3416,"title":"[docs] Add deployment guide section for Logstash","user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/docs","name":"docs","color":"c7def8"}],"state":"open","locked":false,"assignee":{"login":"palecur","id":1779279,"avatar_url":"https://avatars.githubusercontent.com/u/1779279?v=3","gravatar_id":"","url":"https://api.github.com/users/palecur","html_url":"https://github.com/palecur","followers_url":"https://api.github.com/users/palecur/followers","following_url":"https://api.github.com/users/palecur/following{/other_user}","gists_url":"https://api.github.com/users/palecur/gists{/gist_id}","starred_url":"https://api.github.com/users/palecur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palecur/subscriptions","organizations_url":"https://api.github.com/users/palecur/orgs","repos_url":"https://api.github.com/users/palecur/repos","events_url":"https://api.github.com/users/palecur/events{/privacy}","received_events_url":"https://api.github.com/users/palecur/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-06-10T22:46:07Z","updated_at":"2015-06-10T22:52:01Z","closed_at":null,"body":"Add a multi-chapter deployment reference guide for Logstash docs. This should cover both scaling for throughput and also for achieving high availability. This doc should start with a simple scenario to deploy LS using a file input and ES as sample output.  We can then introduce more complex deployment architectures using diagrams. Specifically it should cover:\r\n\r\n- [ ] Simple 1 instance Logstash with file input and ES output, no filters. We should assume this to be a clustered ES setup and use the `http` protocol to index events\r\n- [ ] Same setup as above, but we introduce filters -- maybe grok filter. We can then document the use of multiple workers to increase throughput (using `-w 8` flag)\r\n- [ ] Introduce LSF as a light weight shipper which collects logs from edge nodes and ships to centralized LS server. \r\n- [ ] Assume user has horizontally scaled ES. Talk about following configuration options:\r\n   - [ ] Manual list of nodes in ES output `hosts` options. We round robin on each request for HA\r\n   - [ ] Introduce hardware load balancer which talks to multiple ES client nodes\r\n   - [ ] Introduce transport client with sniffing option\r\n- [ ] Scale out Logstash throughput by introducing parallel pipelines (by starting new LS instances), each sharing the workload or processing completely separate configs (file, twitter, etc)\r\n- [ ] Introduce the use of message brokers to split the pipeline into shippers and indexers phases. List available options like Redis, Kafka, RabbitMQ\r\n- [ ] High availability Logstash setup using a load balancer in front of the inputs. Also talk about instances going down and how to monitor them (using heartbeat plugin for now)\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/110940287","html_url":"https://github.com/elastic/logstash/issues/3416#issuecomment-110940287","issue_url":"https://api.github.com/repos/elastic/logstash/issues/3416","id":110940287,"user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"created_at":"2015-06-10T22:52:01Z","updated_at":"2015-06-10T22:52:01Z","body":"/cc @tbragin @palecur "}}