{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/32","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/32/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/32/events","html_url":"https://github.com/logstash-plugins/logstash-input-kafka/issues/32","id":70810987,"number":32,"title":"Consumer group not reading from all partitions?","user":{"login":"mheese","id":1955729,"avatar_url":"https://avatars.githubusercontent.com/u/1955729?v=3","gravatar_id":"","url":"https://api.github.com/users/mheese","html_url":"https://github.com/mheese","followers_url":"https://api.github.com/users/mheese/followers","following_url":"https://api.github.com/users/mheese/following{/other_user}","gists_url":"https://api.github.com/users/mheese/gists{/gist_id}","starred_url":"https://api.github.com/users/mheese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mheese/subscriptions","organizations_url":"https://api.github.com/users/mheese/orgs","repos_url":"https://api.github.com/users/mheese/repos","events_url":"https://api.github.com/users/mheese/events{/privacy}","received_events_url":"https://api.github.com/users/mheese/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-24T22:32:09Z","updated_at":"2015-04-24T23:15:21Z","closed_at":null,"body":"Hi there,\r\n\r\nI have a setup with a kafka topic that has 6 partitions and 2 different logstash servers forming a consumer group of that topic and basically just outputting it to ElasticSearch. I configured the input currently with 3 workers on each server and a unique consumer group name.\r\n\r\nThe setup works fine for a few days. After that, however, it apparently is not consuming from all partitions anymore. I can see gaps in ElasticSearch that are pretty consistent with missing messages from certain partitions. I'm not a 100% sure if that is the issue, however, as the producer to this topic is not producing with a round-robin partitioner (see https://github.com/logstash-plugins/logstash-output-kafka/issues/17), it is hard to tell.\r\n\r\nPlease let me know what information you need for debugging this issue. I can provide complete logstash configs if that would be helpful."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/comments/96093660","html_url":"https://github.com/logstash-plugins/logstash-input-kafka/issues/32#issuecomment-96093660","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kafka/issues/32","id":96093660,"user":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"created_at":"2015-04-24T23:15:21Z","updated_at":"2015-04-24T23:15:21Z","body":"Just to be clear:\r\n\r\nyou have 2 logstash agents running on separate machines. each is running 3 kafka inputs, where each input has a unique consumer group, different from the other two inputs?\r\n\r\nTo split a topic's messages across each of the workers/logstash instances, it is ideal that all belong to the same consumer-group.\r\n\r\nThe kafka input plugin leverages the official high-level consumer in Kafka, so it is already threaded for you. Not sure you need to go ahead and launch multiple input plugins on each logstash instance.\r\n\r\nAm I understanding the topology / does that make sense?"}}