{"action":"created","comment":{"url":"https://api.github.com/repos/Parsely/pykafka/pulls/comments/37469476","id":37469476,"diff_hunk":"@@ -83,17 +75,34 @@ def __init__(self,\n         :param retry_backoff_ms: The amount of time (in milliseconds) to\n             back off during produce request retries.\n         :type retry_backoff_ms: int\n-        :param required_acks: How many other brokers must have committed the\n-            data to their log and acknowledged this to the leader before a\n-            request is considered complete?\n+        :param required_acks: The number of other brokers must have committed","path":"pykafka/producer.py","position":89,"original_position":89,"commit_id":"790e2daeae59535ecef2f813942f42182df0136b","original_commit_id":"790e2daeae59535ecef2f813942f42182df0136b","user":{"login":"yungchin","id":780001,"avatar_url":"https://avatars.githubusercontent.com/u/780001?v=3","gravatar_id":"","url":"https://api.github.com/users/yungchin","html_url":"https://github.com/yungchin","followers_url":"https://api.github.com/users/yungchin/followers","following_url":"https://api.github.com/users/yungchin/following{/other_user}","gists_url":"https://api.github.com/users/yungchin/gists{/gist_id}","starred_url":"https://api.github.com/users/yungchin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yungchin/subscriptions","organizations_url":"https://api.github.com/users/yungchin/orgs","repos_url":"https://api.github.com/users/yungchin/repos","events_url":"https://api.github.com/users/yungchin/events{/privacy}","received_events_url":"https://api.github.com/users/yungchin/received_events","type":"User","site_admin":false},"body":"Probably missing a \"that\" before \"must\"","created_at":"2015-08-19T21:18:37Z","updated_at":"2015-08-19T21:18:37Z","html_url":"https://github.com/Parsely/pykafka/pull/177#discussion_r37469476","pull_request_url":"https://api.github.com/repos/Parsely/pykafka/pulls/177","_links":{"self":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/comments/37469476"},"html":{"href":"https://github.com/Parsely/pykafka/pull/177#discussion_r37469476"},"pull_request":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/177"}}},"pull_request":{"url":"https://api.github.com/repos/Parsely/pykafka/pulls/177","id":36883989,"html_url":"https://github.com/Parsely/pykafka/pull/177","diff_url":"https://github.com/Parsely/pykafka/pull/177.diff","patch_url":"https://github.com/Parsely/pykafka/pull/177.patch","issue_url":"https://api.github.com/repos/Parsely/pykafka/issues/177","number":177,"state":"open","locked":false,"title":"Asynchronous Producer","user":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"body":"This pull request implements `AsyncProducer`, which uses a thread per broker instead of the main thread for all brokers.\r\n\r\nCode that can be used to test this producer's throughput:\r\n\r\n```python\r\nclient = KafkaClient(hosts='parsely.kafka.hosts:9092')\r\ntopic = client.topics['emmett.test.async']\r\n\r\nproducer = topic.get_producer(\r\n    compression=CompressionType.SNAPPY\r\n)\r\n\r\nstart = time.time()\r\nnext_print = start + 5\r\ncount = 0\r\nbatch_range = 20\r\nwhile True:\r\n    producer.produce([\"msg {} {}\".format(count, i) for i in xrange(batch_range)])\r\n    count += batch_range\r\n    now = time.time()\r\n    if now > next_print:\r\n        dur = now - start\r\n        print '{} messages per second'.format(count / dur)\r\n        count = 0\r\n        start = time.time()\r\n        next_print = start + 5\r\n```\r\n\r\nThroughput test results with the above script (how many times per second each producer can return from `produce()`):\r\nSynchronous producer:\r\n``` \r\n66.2043259773 messages per second\r\n66.260400353 messages per second\r\n66.244598007 messages per second\r\n66.3166715681 messages per second\r\n```\r\nAsync producer:\r\n```\r\n17960.6287112 messages per second\r\n16711.675672 messages per second\r\n18904.4348297 messages per second\r\n18581.195363 messages per second\r\n19335.5599431 messages per second\r\n17676.1641205 messages per second\r\n22010.9735254 messages per second\r\n20010.7615108 messages per second\r\n```\r\n\r\nThe old synchronous producer was doing about 4500 messages per second.\r\n\r\nConnects to #124.\r\n\r\nTodo:\r\n*  [x] Monitor thread death\r\n*  [x] Abstract thread management logic into the `QueueInfo` class\r\n*  [x] Make `Producer` as small as possible by having it be a special case of `AsyncProducer`\r\n*  [x] Tests","created_at":"2015-06-03T22:13:03Z","updated_at":"2015-08-19T21:18:37Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4b32342921f164bb55110c77e3dba6d49def0319","assignee":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Parsely/pykafka/milestones/5","html_url":"https://github.com/Parsely/pykafka/milestones/1.2.0","labels_url":"https://api.github.com/repos/Parsely/pykafka/milestones/5/labels","id":1141520,"number":5,"title":"1.2.0","description":"Release including librdkafka, and asynchronous producer, and probably some other new stuff.","creator":{"login":"kbourgoin","id":136574,"avatar_url":"https://avatars.githubusercontent.com/u/136574?v=3","gravatar_id":"","url":"https://api.github.com/users/kbourgoin","html_url":"https://github.com/kbourgoin","followers_url":"https://api.github.com/users/kbourgoin/followers","following_url":"https://api.github.com/users/kbourgoin/following{/other_user}","gists_url":"https://api.github.com/users/kbourgoin/gists{/gist_id}","starred_url":"https://api.github.com/users/kbourgoin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbourgoin/subscriptions","organizations_url":"https://api.github.com/users/kbourgoin/orgs","repos_url":"https://api.github.com/users/kbourgoin/repos","events_url":"https://api.github.com/users/kbourgoin/events{/privacy}","received_events_url":"https://api.github.com/users/kbourgoin/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":4,"state":"open","created_at":"2015-05-31T03:18:21Z","updated_at":"2015-07-23T23:01:14Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/Parsely/pykafka/pulls/177/commits","review_comments_url":"https://api.github.com/repos/Parsely/pykafka/pulls/177/comments","review_comment_url":"https://api.github.com/repos/Parsely/pykafka/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Parsely/pykafka/issues/177/comments","statuses_url":"https://api.github.com/repos/Parsely/pykafka/statuses/790e2daeae59535ecef2f813942f42182df0136b","head":{"label":"Parsely:feature/async-producer","ref":"feature/async-producer","sha":"790e2daeae59535ecef2f813942f42182df0136b","user":{"login":"Parsely","id":779641,"avatar_url":"https://avatars.githubusercontent.com/u/779641?v=3","gravatar_id":"","url":"https://api.github.com/users/Parsely","html_url":"https://github.com/Parsely","followers_url":"https://api.github.com/users/Parsely/followers","following_url":"https://api.github.com/users/Parsely/following{/other_user}","gists_url":"https://api.github.com/users/Parsely/gists{/gist_id}","starred_url":"https://api.github.com/users/Parsely/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Parsely/subscriptions","organizations_url":"https://api.github.com/users/Parsely/orgs","repos_url":"https://api.github.com/users/Parsely/repos","events_url":"https://api.github.com/users/Parsely/events{/privacy}","received_events_url":"https://api.github.com/users/Parsely/received_events","type":"Organization","site_admin":false},"repo":{"id":4978045,"name":"pykafka","full_name":"Parsely/pykafka","owner":{"login":"Parsely","id":779641,"avatar_url":"https://avatars.githubusercontent.com/u/779641?v=3","gravatar_id":"","url":"https://api.github.com/users/Parsely","html_url":"https://github.com/Parsely","followers_url":"https://api.github.com/users/Parsely/followers","following_url":"https://api.github.com/users/Parsely/following{/other_user}","gists_url":"https://api.github.com/users/Parsely/gists{/gist_id}","starred_url":"https://api.github.com/users/Parsely/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Parsely/subscriptions","organizations_url":"https://api.github.com/users/Parsely/orgs","repos_url":"https://api.github.com/users/Parsely/repos","events_url":"https://api.github.com/users/Parsely/events{/privacy}","received_events_url":"https://api.github.com/users/Parsely/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Parsely/pykafka","description":"Kafka client for Python","fork":false,"url":"https://api.github.com/repos/Parsely/pykafka","forks_url":"https://api.github.com/repos/Parsely/pykafka/forks","keys_url":"https://api.github.com/repos/Parsely/pykafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Parsely/pykafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Parsely/pykafka/teams","hooks_url":"https://api.github.com/repos/Parsely/pykafka/hooks","issue_events_url":"https://api.github.com/repos/Parsely/pykafka/issues/events{/number}","events_url":"https://api.github.com/repos/Parsely/pykafka/events","assignees_url":"https://api.github.com/repos/Parsely/pykafka/assignees{/user}","branches_url":"https://api.github.com/repos/Parsely/pykafka/branches{/branch}","tags_url":"https://api.github.com/repos/Parsely/pykafka/tags","blobs_url":"https://api.github.com/repos/Parsely/pykafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Parsely/pykafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Parsely/pykafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/Parsely/pykafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Parsely/pykafka/statuses/{sha}","languages_url":"https://api.github.com/repos/Parsely/pykafka/languages","stargazers_url":"https://api.github.com/repos/Parsely/pykafka/stargazers","contributors_url":"https://api.github.com/repos/Parsely/pykafka/contributors","subscribers_url":"https://api.github.com/repos/Parsely/pykafka/subscribers","subscription_url":"https://api.github.com/repos/Parsely/pykafka/subscription","commits_url":"https://api.github.com/repos/Parsely/pykafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/Parsely/pykafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/Parsely/pykafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/Parsely/pykafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/Parsely/pykafka/contents/{+path}","compare_url":"https://api.github.com/repos/Parsely/pykafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Parsely/pykafka/merges","archive_url":"https://api.github.com/repos/Parsely/pykafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Parsely/pykafka/downloads","issues_url":"https://api.github.com/repos/Parsely/pykafka/issues{/number}","pulls_url":"https://api.github.com/repos/Parsely/pykafka/pulls{/number}","milestones_url":"https://api.github.com/repos/Parsely/pykafka/milestones{/number}","notifications_url":"https://api.github.com/repos/Parsely/pykafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Parsely/pykafka/labels{/name}","releases_url":"https://api.github.com/repos/Parsely/pykafka/releases{/id}","created_at":"2012-07-10T18:02:21Z","updated_at":"2015-08-17T07:26:44Z","pushed_at":"2015-08-19T20:32:51Z","git_url":"git://github.com/Parsely/pykafka.git","ssh_url":"git@github.com:Parsely/pykafka.git","clone_url":"https://github.com/Parsely/pykafka.git","svn_url":"https://github.com/Parsely/pykafka","homepage":"http://pykafka.readthedocs.org/","size":3329,"stargazers_count":154,"watchers_count":154,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":42,"mirror_url":null,"open_issues_count":18,"forks":42,"open_issues":18,"watchers":154,"default_branch":"master"}},"base":{"label":"Parsely:master","ref":"master","sha":"dd6997e9ee2ebe56e7c612de757fccd2a4c2f367","user":{"login":"Parsely","id":779641,"avatar_url":"https://avatars.githubusercontent.com/u/779641?v=3","gravatar_id":"","url":"https://api.github.com/users/Parsely","html_url":"https://github.com/Parsely","followers_url":"https://api.github.com/users/Parsely/followers","following_url":"https://api.github.com/users/Parsely/following{/other_user}","gists_url":"https://api.github.com/users/Parsely/gists{/gist_id}","starred_url":"https://api.github.com/users/Parsely/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Parsely/subscriptions","organizations_url":"https://api.github.com/users/Parsely/orgs","repos_url":"https://api.github.com/users/Parsely/repos","events_url":"https://api.github.com/users/Parsely/events{/privacy}","received_events_url":"https://api.github.com/users/Parsely/received_events","type":"Organization","site_admin":false},"repo":{"id":4978045,"name":"pykafka","full_name":"Parsely/pykafka","owner":{"login":"Parsely","id":779641,"avatar_url":"https://avatars.githubusercontent.com/u/779641?v=3","gravatar_id":"","url":"https://api.github.com/users/Parsely","html_url":"https://github.com/Parsely","followers_url":"https://api.github.com/users/Parsely/followers","following_url":"https://api.github.com/users/Parsely/following{/other_user}","gists_url":"https://api.github.com/users/Parsely/gists{/gist_id}","starred_url":"https://api.github.com/users/Parsely/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Parsely/subscriptions","organizations_url":"https://api.github.com/users/Parsely/orgs","repos_url":"https://api.github.com/users/Parsely/repos","events_url":"https://api.github.com/users/Parsely/events{/privacy}","received_events_url":"https://api.github.com/users/Parsely/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Parsely/pykafka","description":"Kafka client for Python","fork":false,"url":"https://api.github.com/repos/Parsely/pykafka","forks_url":"https://api.github.com/repos/Parsely/pykafka/forks","keys_url":"https://api.github.com/repos/Parsely/pykafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Parsely/pykafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Parsely/pykafka/teams","hooks_url":"https://api.github.com/repos/Parsely/pykafka/hooks","issue_events_url":"https://api.github.com/repos/Parsely/pykafka/issues/events{/number}","events_url":"https://api.github.com/repos/Parsely/pykafka/events","assignees_url":"https://api.github.com/repos/Parsely/pykafka/assignees{/user}","branches_url":"https://api.github.com/repos/Parsely/pykafka/branches{/branch}","tags_url":"https://api.github.com/repos/Parsely/pykafka/tags","blobs_url":"https://api.github.com/repos/Parsely/pykafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Parsely/pykafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Parsely/pykafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/Parsely/pykafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Parsely/pykafka/statuses/{sha}","languages_url":"https://api.github.com/repos/Parsely/pykafka/languages","stargazers_url":"https://api.github.com/repos/Parsely/pykafka/stargazers","contributors_url":"https://api.github.com/repos/Parsely/pykafka/contributors","subscribers_url":"https://api.github.com/repos/Parsely/pykafka/subscribers","subscription_url":"https://api.github.com/repos/Parsely/pykafka/subscription","commits_url":"https://api.github.com/repos/Parsely/pykafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/Parsely/pykafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/Parsely/pykafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/Parsely/pykafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/Parsely/pykafka/contents/{+path}","compare_url":"https://api.github.com/repos/Parsely/pykafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Parsely/pykafka/merges","archive_url":"https://api.github.com/repos/Parsely/pykafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Parsely/pykafka/downloads","issues_url":"https://api.github.com/repos/Parsely/pykafka/issues{/number}","pulls_url":"https://api.github.com/repos/Parsely/pykafka/pulls{/number}","milestones_url":"https://api.github.com/repos/Parsely/pykafka/milestones{/number}","notifications_url":"https://api.github.com/repos/Parsely/pykafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Parsely/pykafka/labels{/name}","releases_url":"https://api.github.com/repos/Parsely/pykafka/releases{/id}","created_at":"2012-07-10T18:02:21Z","updated_at":"2015-08-17T07:26:44Z","pushed_at":"2015-08-19T20:32:51Z","git_url":"git://github.com/Parsely/pykafka.git","ssh_url":"git@github.com:Parsely/pykafka.git","clone_url":"https://github.com/Parsely/pykafka.git","svn_url":"https://github.com/Parsely/pykafka","homepage":"http://pykafka.readthedocs.org/","size":3329,"stargazers_count":154,"watchers_count":154,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":42,"mirror_url":null,"open_issues_count":18,"forks":42,"open_issues":18,"watchers":154,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/177"},"html":{"href":"https://github.com/Parsely/pykafka/pull/177"},"issue":{"href":"https://api.github.com/repos/Parsely/pykafka/issues/177"},"comments":{"href":"https://api.github.com/repos/Parsely/pykafka/issues/177/comments"},"review_comments":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/177/comments"},"review_comment":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Parsely/pykafka/pulls/177/commits"},"statuses":{"href":"https://api.github.com/repos/Parsely/pykafka/statuses/790e2daeae59535ecef2f813942f42182df0136b"}}}}