{"action":"created","issue":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/312","labels_url":"https://api.github.com/repos/Parsely/pykafka/issues/312/labels{/name}","comments_url":"https://api.github.com/repos/Parsely/pykafka/issues/312/comments","events_url":"https://api.github.com/repos/Parsely/pykafka/issues/312/events","html_url":"https://github.com/Parsely/pykafka/issues/312","id":113051090,"number":312,"title":"PyKafka NoMessagesConsumed Errors","user":{"login":"jasonrhaas","id":7018237,"avatar_url":"https://avatars.githubusercontent.com/u/7018237?v=3","gravatar_id":"","url":"https://api.github.com/users/jasonrhaas","html_url":"https://github.com/jasonrhaas","followers_url":"https://api.github.com/users/jasonrhaas/followers","following_url":"https://api.github.com/users/jasonrhaas/following{/other_user}","gists_url":"https://api.github.com/users/jasonrhaas/gists{/gist_id}","starred_url":"https://api.github.com/users/jasonrhaas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasonrhaas/subscriptions","organizations_url":"https://api.github.com/users/jasonrhaas/orgs","repos_url":"https://api.github.com/users/jasonrhaas/repos","events_url":"https://api.github.com/users/jasonrhaas/events{/privacy}","received_events_url":"https://api.github.com/users/jasonrhaas/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Parsely/pykafka/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"yungchin","id":780001,"avatar_url":"https://avatars.githubusercontent.com/u/780001?v=3","gravatar_id":"","url":"https://api.github.com/users/yungchin","html_url":"https://github.com/yungchin","followers_url":"https://api.github.com/users/yungchin/followers","following_url":"https://api.github.com/users/yungchin/following{/other_user}","gists_url":"https://api.github.com/users/yungchin/gists{/gist_id}","starred_url":"https://api.github.com/users/yungchin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yungchin/subscriptions","organizations_url":"https://api.github.com/users/yungchin/orgs","repos_url":"https://api.github.com/users/yungchin/repos","events_url":"https://api.github.com/users/yungchin/events{/privacy}","received_events_url":"https://api.github.com/users/yungchin/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-10-23T16:10:00Z","updated_at":"2015-10-26T14:25:18Z","closed_at":null,"body":"PyKafka throws an error occasionally when trying to iterate through messages in a Kafka queue.  I imagine the issue is some consumer or zookeeper timeout issue, since I know that I am not consuming faster than my Kafka queue is being produced to.  I made a simple script to manually iterate through messages and after a while it throws this error:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"pykafka_test.py\", line 50, in <module>\r\n    k.consume()\r\n  File \"pykafka_test.py\", line 37, in consume\r\n    print self.balanced_consumer.consume().value\r\n  File \"/usr/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py\", line 575, in consume\r\n    message = self._consumer.consume(block=block)\r\n  File \"/usr/local/lib/python2.7/site-packages/pykafka/simpleconsumer.py\", line 345, in consume\r\n    self._raise_worker_exceptions()\r\n  File \"/usr/local/lib/python2.7/site-packages/pykafka/simpleconsumer.py\", line 199, in _raise_worker_exceptions\r\n    raise ex\r\npykafka.exceptions.NoMessagesConsumedError\r\n```\r\n\r\nI imagine its seeing some *null* value or something and just raising that exception.  Is this the desired behavior?  Should it return `None` instead?  In a gist that @amontalenti wrote, he had a code block that did this:\r\n\r\n```\r\n        msg = self.consumer.consume()\r\n        if not msg:\r\n            # nothing here\r\n            return\r\n```\r\n\r\nThis would work if `self.consumer.consume()` was returning `None`, but instead it raises an error.  You can catch this error with `except: NoMessagesConsumed` but I think this error message may be misleading and something else is going on.\r\n\r\nHere's a simple script I'm using to test this:\r\n\r\nhttps://gist.github.com/jasonrhaas/2281296a538908cfa85b"},"comment":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/comments/151153970","html_url":"https://github.com/Parsely/pykafka/issues/312#issuecomment-151153970","issue_url":"https://api.github.com/repos/Parsely/pykafka/issues/312","id":151153970,"user":{"login":"yungchin","id":780001,"avatar_url":"https://avatars.githubusercontent.com/u/780001?v=3","gravatar_id":"","url":"https://api.github.com/users/yungchin","html_url":"https://github.com/yungchin","followers_url":"https://api.github.com/users/yungchin/followers","following_url":"https://api.github.com/users/yungchin/following{/other_user}","gists_url":"https://api.github.com/users/yungchin/gists{/gist_id}","starred_url":"https://api.github.com/users/yungchin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yungchin/subscriptions","organizations_url":"https://api.github.com/users/yungchin/orgs","repos_url":"https://api.github.com/users/yungchin/repos","events_url":"https://api.github.com/users/yungchin/events{/privacy}","received_events_url":"https://api.github.com/users/yungchin/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T14:25:18Z","updated_at":"2015-10-26T14:25:18Z","body":"Thanks very much for this. This exception was introduced with ac5fda0d, and indicates that a truncated message was received from kafka. We can probably change how we handle the error (as you've found, it currently aborts the fetcher thread).\r\n\r\nI'm hoping @emmett9001 (who wrote the commit I referenced) may be able provide some extra context: would this error be recoverable if we re-fetch from the same offset? That is, is it just a truncated network transmission, or is there really only a partial message stored in kafka?"}}