{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/421","labels_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/421/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/421/comments","events_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/421/events","html_url":"https://github.com/elastic/logstash-forwarder/issues/421","id":60444026,"number":421,"title":"Repeated \"Read Error looking for ACK\" Error Messages","user":{"login":"ninjada","id":7445966,"avatar_url":"https://avatars.githubusercontent.com/u/7445966?v=3","gravatar_id":"","url":"https://api.github.com/users/ninjada","html_url":"https://github.com/ninjada","followers_url":"https://api.github.com/users/ninjada/followers","following_url":"https://api.github.com/users/ninjada/following{/other_user}","gists_url":"https://api.github.com/users/ninjada/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjada/subscriptions","organizations_url":"https://api.github.com/users/ninjada/orgs","repos_url":"https://api.github.com/users/ninjada/repos","events_url":"https://api.github.com/users/ninjada/events{/privacy}","received_events_url":"https://api.github.com/users/ninjada/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-10T03:03:01Z","updated_at":"2015-08-19T20:35:44Z","closed_at":"2015-08-19T20:35:44Z","body":"Hi,\r\n\r\nMy logstash staging environment was running ok for sometime. However, after scaling up the number of servers and thus logs being harvested in testing for a pre-production run, I've started seeing repeated read errors on logstash-forwarder's log. \r\n\r\nAlso, after some time, the logstash and elasticsearch server will crash with java heapsize errors.. \r\n\r\nNote: atm, i'm running the ELK stack on one node, but it is a big server (32GB RAM) and there are only 4-5 servers pushing logs to it via logstash-forwarder..\r\n\r\nLogstash-forwarder will report this, generally at the same time on all forwarding servers:\r\n \r\n2015/03/10 13:33:54.468798 Read error looking for ack: read tcp 192.168.0.100:5000: i/o timeout\r\n\r\nIt'll wait for some time, and begin processing again, unless the ELK stack falls over at some level..\r\n\r\nan example of my logstash-forwarder config and command line:\r\n\r\ncmd line: -config /etc/logstash-forwarder -spool-size 100'\r\n\r\nconfig: https://gist.github.com/ninjada/43a5d3d5485526518c8e\r\n\r\nThose are the main types of logs, aside from that theres 2 servers harvesting varnishncsa.log which are the most intensive.\r\n\r\nI've been running logstash-forwarder-0.4.0-1 built myself about 3-4 weeks ago. I've just built the latest version as of this morning (same version, but has a fair few changes) and tried it on one of the servers\r\n\r\nIOwait on the logstash server seems fine, so I can't seem to nail down what's causing this. \r\n\r\nAny ideas?\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/comments/132773742","html_url":"https://github.com/elastic/logstash-forwarder/issues/421#issuecomment-132773742","issue_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/421","id":132773742,"user":{"login":"jordansissel","id":131818,"avatar_url":"https://avatars.githubusercontent.com/u/131818?v=3","gravatar_id":"","url":"https://api.github.com/users/jordansissel","html_url":"https://github.com/jordansissel","followers_url":"https://api.github.com/users/jordansissel/followers","following_url":"https://api.github.com/users/jordansissel/following{/other_user}","gists_url":"https://api.github.com/users/jordansissel/gists{/gist_id}","starred_url":"https://api.github.com/users/jordansissel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordansissel/subscriptions","organizations_url":"https://api.github.com/users/jordansissel/orgs","repos_url":"https://api.github.com/users/jordansissel/repos","events_url":"https://api.github.com/users/jordansissel/events{/privacy}","received_events_url":"https://api.github.com/users/jordansissel/received_events","type":"User","site_admin":false},"created_at":"2015-08-19T20:35:44Z","updated_at":"2015-08-19T20:35:44Z","body":"This kind of thing is probably best to ask on discuss.elastic.co - sorry for the delays in responding.\r\n\r\n\"Read error waiting for ack\" means taht lsf gave upw aiting for Logstash to acknowledge successful handling of the current batch of events. This usually means something downstream of lsf (network, logstash, elasticsearch, etc) is down or unable to keep up with your event throughput.\r\n\r\nLet's follow up on https://discuss.elastic.co/ with some more details about your entire stack (lsf/logstash/es/disk/network/etc) and hopefuly we can move you forward from there if you're still having troubles."}}