{"action":"opened","number":335,"pull_request":{"url":"https://api.github.com/repos/mumrah/kafka-python/pulls/335","id":30370350,"html_url":"https://github.com/mumrah/kafka-python/pull/335","diff_url":"https://github.com/mumrah/kafka-python/pull/335.diff","patch_url":"https://github.com/mumrah/kafka-python/pull/335.patch","issue_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335","number":335,"state":"open","locked":false,"title":"Wrong partitions distribution logic for MP Consumer","user":{"login":"vshlapakov","id":448111,"avatar_url":"https://avatars.githubusercontent.com/u/448111?v=3","gravatar_id":"","url":"https://api.github.com/users/vshlapakov","html_url":"https://github.com/vshlapakov","followers_url":"https://api.github.com/users/vshlapakov/followers","following_url":"https://api.github.com/users/vshlapakov/following{/other_user}","gists_url":"https://api.github.com/users/vshlapakov/gists{/gist_id}","starred_url":"https://api.github.com/users/vshlapakov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vshlapakov/subscriptions","organizations_url":"https://api.github.com/users/vshlapakov/orgs","repos_url":"https://api.github.com/users/vshlapakov/repos","events_url":"https://api.github.com/users/vshlapakov/events{/privacy}","received_events_url":"https://api.github.com/users/vshlapakov/received_events","type":"User","site_admin":false},"body":"I could be wrong, but it seems that there's wrong partitions distribution logic for `MultiProcessConsumer`.\r\nIt's not even at all currently, it can be checked with the following function (cut from the code and wrapped):\r\n```python \r\n>>> def get_chunks(partitions, num_procs, partitions_per_proc=0):\r\n...     if not partitions_per_proc:\r\n...             partitions_per_proc = round(len(partitions) * 1.0 / num_procs)\r\n...             if partitions_per_proc < num_procs * 0.5:\r\n...                     partitions_per_proc += 1\r\n...     chunker = lambda *x: [] + list(x)\r\n...     return map(chunker, *[iter(partitions)] * int(partitions_per_proc))\r\n...\r\n```\r\nWrong cases examples:\r\n```python\r\n>>> get_chunks(range(16), 3)\r\n[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, None, None, None, None]]\r\n# there should be 3 chunks!\r\n>>> get_chunks(range(16), 8)\r\n[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 13, 14], [15, None, None]]\r\n# there should be 8 chunks by 2 partitions each\r\n```\r\nThis PR solves this issue by fixing the condition when we should increase `partitions_per_proc` by 1:\r\n```python\r\n>>> get_chunks(range(16), 1)\r\n[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 2)\r\n[[0, 1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 3)\r\n[[0, 1, 2, 3, 4, 5], [6, 7, 8, 9, 10, 11], [12, 13, 14, 15, None, None]]\r\n>>> get_chunks(range(16), 4)\r\n[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 8)\r\n[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]]\r\n```","created_at":"2015-03-03T12:47:01Z","updated_at":"2015-03-03T12:47:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mumrah/kafka-python/pulls/335/commits","review_comments_url":"https://api.github.com/repos/mumrah/kafka-python/pulls/335/comments","review_comment_url":"https://api.github.com/repos/mumrah/kafka-python/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335/comments","statuses_url":"https://api.github.com/repos/mumrah/kafka-python/statuses/ac66fe9159723c6707258bbb4c6727cf66a78219","head":{"label":"scrapinghub:fix-mp-consumer-distribution","ref":"fix-mp-consumer-distribution","sha":"ac66fe9159723c6707258bbb4c6727cf66a78219","user":{"login":"scrapinghub","id":699596,"avatar_url":"https://avatars.githubusercontent.com/u/699596?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapinghub","html_url":"https://github.com/scrapinghub","followers_url":"https://api.github.com/users/scrapinghub/followers","following_url":"https://api.github.com/users/scrapinghub/following{/other_user}","gists_url":"https://api.github.com/users/scrapinghub/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapinghub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapinghub/subscriptions","organizations_url":"https://api.github.com/users/scrapinghub/orgs","repos_url":"https://api.github.com/users/scrapinghub/repos","events_url":"https://api.github.com/users/scrapinghub/events{/privacy}","received_events_url":"https://api.github.com/users/scrapinghub/received_events","type":"Organization","site_admin":false},"repo":{"id":31545075,"name":"kafka-python","full_name":"scrapinghub/kafka-python","owner":{"login":"scrapinghub","id":699596,"avatar_url":"https://avatars.githubusercontent.com/u/699596?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapinghub","html_url":"https://github.com/scrapinghub","followers_url":"https://api.github.com/users/scrapinghub/followers","following_url":"https://api.github.com/users/scrapinghub/following{/other_user}","gists_url":"https://api.github.com/users/scrapinghub/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapinghub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapinghub/subscriptions","organizations_url":"https://api.github.com/users/scrapinghub/orgs","repos_url":"https://api.github.com/users/scrapinghub/repos","events_url":"https://api.github.com/users/scrapinghub/events{/privacy}","received_events_url":"https://api.github.com/users/scrapinghub/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scrapinghub/kafka-python","description":"Python client for Apache Kafka","fork":true,"url":"https://api.github.com/repos/scrapinghub/kafka-python","forks_url":"https://api.github.com/repos/scrapinghub/kafka-python/forks","keys_url":"https://api.github.com/repos/scrapinghub/kafka-python/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scrapinghub/kafka-python/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scrapinghub/kafka-python/teams","hooks_url":"https://api.github.com/repos/scrapinghub/kafka-python/hooks","issue_events_url":"https://api.github.com/repos/scrapinghub/kafka-python/issues/events{/number}","events_url":"https://api.github.com/repos/scrapinghub/kafka-python/events","assignees_url":"https://api.github.com/repos/scrapinghub/kafka-python/assignees{/user}","branches_url":"https://api.github.com/repos/scrapinghub/kafka-python/branches{/branch}","tags_url":"https://api.github.com/repos/scrapinghub/kafka-python/tags","blobs_url":"https://api.github.com/repos/scrapinghub/kafka-python/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scrapinghub/kafka-python/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scrapinghub/kafka-python/git/refs{/sha}","trees_url":"https://api.github.com/repos/scrapinghub/kafka-python/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scrapinghub/kafka-python/statuses/{sha}","languages_url":"https://api.github.com/repos/scrapinghub/kafka-python/languages","stargazers_url":"https://api.github.com/repos/scrapinghub/kafka-python/stargazers","contributors_url":"https://api.github.com/repos/scrapinghub/kafka-python/contributors","subscribers_url":"https://api.github.com/repos/scrapinghub/kafka-python/subscribers","subscription_url":"https://api.github.com/repos/scrapinghub/kafka-python/subscription","commits_url":"https://api.github.com/repos/scrapinghub/kafka-python/commits{/sha}","git_commits_url":"https://api.github.com/repos/scrapinghub/kafka-python/git/commits{/sha}","comments_url":"https://api.github.com/repos/scrapinghub/kafka-python/comments{/number}","issue_comment_url":"https://api.github.com/repos/scrapinghub/kafka-python/issues/comments{/number}","contents_url":"https://api.github.com/repos/scrapinghub/kafka-python/contents/{+path}","compare_url":"https://api.github.com/repos/scrapinghub/kafka-python/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scrapinghub/kafka-python/merges","archive_url":"https://api.github.com/repos/scrapinghub/kafka-python/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scrapinghub/kafka-python/downloads","issues_url":"https://api.github.com/repos/scrapinghub/kafka-python/issues{/number}","pulls_url":"https://api.github.com/repos/scrapinghub/kafka-python/pulls{/number}","milestones_url":"https://api.github.com/repos/scrapinghub/kafka-python/milestones{/number}","notifications_url":"https://api.github.com/repos/scrapinghub/kafka-python/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scrapinghub/kafka-python/labels{/name}","releases_url":"https://api.github.com/repos/scrapinghub/kafka-python/releases{/id}","created_at":"2015-03-02T14:48:39Z","updated_at":"2015-03-02T14:48:39Z","pushed_at":"2015-03-03T12:33:02Z","git_url":"git://github.com/scrapinghub/kafka-python.git","ssh_url":"git@github.com:scrapinghub/kafka-python.git","clone_url":"https://github.com/scrapinghub/kafka-python.git","svn_url":"https://github.com/scrapinghub/kafka-python","homepage":"","size":972,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mumrah:master","ref":"master","sha":"e13eb0de8903541ec30050a8e7dd8a168752cea3","user":{"login":"mumrah","id":55116,"avatar_url":"https://avatars.githubusercontent.com/u/55116?v=3","gravatar_id":"","url":"https://api.github.com/users/mumrah","html_url":"https://github.com/mumrah","followers_url":"https://api.github.com/users/mumrah/followers","following_url":"https://api.github.com/users/mumrah/following{/other_user}","gists_url":"https://api.github.com/users/mumrah/gists{/gist_id}","starred_url":"https://api.github.com/users/mumrah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumrah/subscriptions","organizations_url":"https://api.github.com/users/mumrah/orgs","repos_url":"https://api.github.com/users/mumrah/repos","events_url":"https://api.github.com/users/mumrah/events{/privacy}","received_events_url":"https://api.github.com/users/mumrah/received_events","type":"User","site_admin":false},"repo":{"id":5934517,"name":"kafka-python","full_name":"mumrah/kafka-python","owner":{"login":"mumrah","id":55116,"avatar_url":"https://avatars.githubusercontent.com/u/55116?v=3","gravatar_id":"","url":"https://api.github.com/users/mumrah","html_url":"https://github.com/mumrah","followers_url":"https://api.github.com/users/mumrah/followers","following_url":"https://api.github.com/users/mumrah/following{/other_user}","gists_url":"https://api.github.com/users/mumrah/gists{/gist_id}","starred_url":"https://api.github.com/users/mumrah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumrah/subscriptions","organizations_url":"https://api.github.com/users/mumrah/orgs","repos_url":"https://api.github.com/users/mumrah/repos","events_url":"https://api.github.com/users/mumrah/events{/privacy}","received_events_url":"https://api.github.com/users/mumrah/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mumrah/kafka-python","description":"Python client for Apache Kafka","fork":false,"url":"https://api.github.com/repos/mumrah/kafka-python","forks_url":"https://api.github.com/repos/mumrah/kafka-python/forks","keys_url":"https://api.github.com/repos/mumrah/kafka-python/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mumrah/kafka-python/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mumrah/kafka-python/teams","hooks_url":"https://api.github.com/repos/mumrah/kafka-python/hooks","issue_events_url":"https://api.github.com/repos/mumrah/kafka-python/issues/events{/number}","events_url":"https://api.github.com/repos/mumrah/kafka-python/events","assignees_url":"https://api.github.com/repos/mumrah/kafka-python/assignees{/user}","branches_url":"https://api.github.com/repos/mumrah/kafka-python/branches{/branch}","tags_url":"https://api.github.com/repos/mumrah/kafka-python/tags","blobs_url":"https://api.github.com/repos/mumrah/kafka-python/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mumrah/kafka-python/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mumrah/kafka-python/git/refs{/sha}","trees_url":"https://api.github.com/repos/mumrah/kafka-python/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mumrah/kafka-python/statuses/{sha}","languages_url":"https://api.github.com/repos/mumrah/kafka-python/languages","stargazers_url":"https://api.github.com/repos/mumrah/kafka-python/stargazers","contributors_url":"https://api.github.com/repos/mumrah/kafka-python/contributors","subscribers_url":"https://api.github.com/repos/mumrah/kafka-python/subscribers","subscription_url":"https://api.github.com/repos/mumrah/kafka-python/subscription","commits_url":"https://api.github.com/repos/mumrah/kafka-python/commits{/sha}","git_commits_url":"https://api.github.com/repos/mumrah/kafka-python/git/commits{/sha}","comments_url":"https://api.github.com/repos/mumrah/kafka-python/comments{/number}","issue_comment_url":"https://api.github.com/repos/mumrah/kafka-python/issues/comments{/number}","contents_url":"https://api.github.com/repos/mumrah/kafka-python/contents/{+path}","compare_url":"https://api.github.com/repos/mumrah/kafka-python/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mumrah/kafka-python/merges","archive_url":"https://api.github.com/repos/mumrah/kafka-python/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mumrah/kafka-python/downloads","issues_url":"https://api.github.com/repos/mumrah/kafka-python/issues{/number}","pulls_url":"https://api.github.com/repos/mumrah/kafka-python/pulls{/number}","milestones_url":"https://api.github.com/repos/mumrah/kafka-python/milestones{/number}","notifications_url":"https://api.github.com/repos/mumrah/kafka-python/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mumrah/kafka-python/labels{/name}","releases_url":"https://api.github.com/repos/mumrah/kafka-python/releases{/id}","created_at":"2012-09-24T13:00:26Z","updated_at":"2015-03-03T01:49:07Z","pushed_at":"2015-03-03T01:49:07Z","git_url":"git://github.com/mumrah/kafka-python.git","ssh_url":"git@github.com:mumrah/kafka-python.git","clone_url":"https://github.com/mumrah/kafka-python.git","svn_url":"https://github.com/mumrah/kafka-python","homepage":"","size":3104,"stargazers_count":414,"watchers_count":414,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":200,"mirror_url":null,"open_issues_count":81,"forks":200,"open_issues":81,"watchers":414,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mumrah/kafka-python/pulls/335"},"html":{"href":"https://github.com/mumrah/kafka-python/pull/335"},"issue":{"href":"https://api.github.com/repos/mumrah/kafka-python/issues/335"},"comments":{"href":"https://api.github.com/repos/mumrah/kafka-python/issues/335/comments"},"review_comments":{"href":"https://api.github.com/repos/mumrah/kafka-python/pulls/335/comments"},"review_comment":{"href":"https://api.github.com/repos/mumrah/kafka-python/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mumrah/kafka-python/pulls/335/commits"},"statuses":{"href":"https://api.github.com/repos/mumrah/kafka-python/statuses/ac66fe9159723c6707258bbb4c6727cf66a78219"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":1,"deletions":1,"changed_files":1}}