{"action":"created","issue":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/194","labels_url":"https://api.github.com/repos/Parsely/pykafka/issues/194/labels{/name}","comments_url":"https://api.github.com/repos/Parsely/pykafka/issues/194/comments","events_url":"https://api.github.com/repos/Parsely/pykafka/issues/194/events","html_url":"https://github.com/Parsely/pykafka/pull/194","id":91385942,"number":194,"title":"public interface for offset resetting","user":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-06-26T23:33:48Z","updated_at":"2015-07-10T22:32:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Parsely/pykafka/pulls/194","html_url":"https://github.com/Parsely/pykafka/pull/194","diff_url":"https://github.com/Parsely/pykafka/pull/194.diff","patch_url":"https://github.com/Parsely/pykafka/pull/194.patch"},"body":"This pull request adds a public interface to the consumers that allows client code to set the current partition offsets to whatever they choose. Previously, this had to be done via a semi-public undocumented interface as detailed in [this issue](https://github.com/Parsely/pykafka/issues/187).\r\n\r\nGiven these changes, the new workflow for specifying offsets from which to consume looks like this:\r\n```python\r\nconsumer = topic.get_simple_consumer(consumer_group=\"testgroup\")\r\npartition_offset_pairs = [(p, get_offset_for_partition(p)) for p in consumer.partitions.itervalues()]\r\nconsumer.reset_offsets(partition_offsets=partition_offset_pairs)\r\n```\r\nThis pull request also ties `reset_offsets` to the `fetch` locking mechanism so that a partition can never be fetching and resetting its offsets at the same time. This makes the unlocking logic at the end of `fetch()` notably more complicated, since one of its error conditions involves calling `reset_offsets`.\r\n\r\n**The following has been resolved in this pull request**\r\n~~The only issue I see with this method is that the consumer has a chance to start fetching messages between `get_simple_consumer` and `reset_offsets`. To get around this,  we could accept the `partition_offset_pairs` in `get_simple_consumer`, but that would require some way to obtain a list of partitions before actually instantiating a consumer. This sounds like a road I'd rather not go down, so the question is: how bad is it that it's impossible to start the consumer at an arbitrary offset that's not the head or tail of the partition? My gut tells me it doesn't matter and this solution is ok.~~\r\nLet me know what you think, @kbourgoin and @yungchin."},"comment":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/comments/120542437","html_url":"https://github.com/Parsely/pykafka/pull/194#issuecomment-120542437","issue_url":"https://api.github.com/repos/Parsely/pykafka/issues/194","id":120542437,"user":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"created_at":"2015-07-10T22:32:27Z","updated_at":"2015-07-10T22:32:27Z","body":"This is now ready for review from @yungchin and @kbourgoin. @yungchin, this hasn't changed a ton since you looked at it, but I fixed some deadlocks and improved the interface for user-supplied offsets."}}