{"action":"created","issue":{"url":"https://api.github.com/repos/mumrah/kafka-python/issues/335","labels_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335/labels{/name}","comments_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335/comments","events_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335/events","html_url":"https://github.com/mumrah/kafka-python/pull/335","id":59636784,"number":335,"title":"Wrong partitions distribution logic for MP Consumer","user":{"login":"vshlapakov","id":448111,"avatar_url":"https://avatars.githubusercontent.com/u/448111?v=3","gravatar_id":"","url":"https://api.github.com/users/vshlapakov","html_url":"https://github.com/vshlapakov","followers_url":"https://api.github.com/users/vshlapakov/followers","following_url":"https://api.github.com/users/vshlapakov/following{/other_user}","gists_url":"https://api.github.com/users/vshlapakov/gists{/gist_id}","starred_url":"https://api.github.com/users/vshlapakov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vshlapakov/subscriptions","organizations_url":"https://api.github.com/users/vshlapakov/orgs","repos_url":"https://api.github.com/users/vshlapakov/repos","events_url":"https://api.github.com/users/vshlapakov/events{/privacy}","received_events_url":"https://api.github.com/users/vshlapakov/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mumrah/kafka-python/labels/consumer","name":"consumer","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-03T12:47:01Z","updated_at":"2015-03-12T09:34:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mumrah/kafka-python/pulls/335","html_url":"https://github.com/mumrah/kafka-python/pull/335","diff_url":"https://github.com/mumrah/kafka-python/pull/335.diff","patch_url":"https://github.com/mumrah/kafka-python/pull/335.patch"},"body":"I could be wrong, but it seems that there's wrong partitions distribution logic for `MultiProcessConsumer`.\r\nIt's not even at all currently, it can be checked with the following function (cut from the code and wrapped):\r\n```python \r\n>>> def get_chunks(partitions, num_procs, partitions_per_proc=0):\r\n...     if not partitions_per_proc:\r\n...             partitions_per_proc = round(len(partitions) * 1.0 / num_procs)\r\n...             if partitions_per_proc < num_procs * 0.5:\r\n...                     partitions_per_proc += 1\r\n...     chunker = lambda *x: [] + list(x)\r\n...     return map(chunker, *[iter(partitions)] * int(partitions_per_proc))\r\n...\r\n```\r\nWrong cases examples:\r\n```python\r\n>>> get_chunks(range(16), 3)\r\n[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, None, None, None, None]]\r\n# there should be 3 chunks!\r\n>>> get_chunks(range(16), 8)\r\n[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 13, 14], [15, None, None]]\r\n# there should be 8 chunks by 2 partitions each\r\n```\r\nThis PR solves this issue by fixing the condition when we should increase `partitions_per_proc` by 1:\r\n```python\r\n>>> get_chunks(range(16), 1)\r\n[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 2)\r\n[[0, 1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 3)\r\n[[0, 1, 2, 3, 4, 5], [6, 7, 8, 9, 10, 11], [12, 13, 14, 15, None, None]]\r\n>>> get_chunks(range(16), 4)\r\n[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]]\r\n>>> get_chunks(range(16), 8)\r\n[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]]\r\n```"},"comment":{"url":"https://api.github.com/repos/mumrah/kafka-python/issues/comments/78448193","html_url":"https://github.com/mumrah/kafka-python/pull/335#issuecomment-78448193","issue_url":"https://api.github.com/repos/mumrah/kafka-python/issues/335","id":78448193,"user":{"login":"vshlapakov","id":448111,"avatar_url":"https://avatars.githubusercontent.com/u/448111?v=3","gravatar_id":"","url":"https://api.github.com/users/vshlapakov","html_url":"https://github.com/vshlapakov","followers_url":"https://api.github.com/users/vshlapakov/followers","following_url":"https://api.github.com/users/vshlapakov/following{/other_user}","gists_url":"https://api.github.com/users/vshlapakov/gists{/gist_id}","starred_url":"https://api.github.com/users/vshlapakov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vshlapakov/subscriptions","organizations_url":"https://api.github.com/users/vshlapakov/orgs","repos_url":"https://api.github.com/users/vshlapakov/repos","events_url":"https://api.github.com/users/vshlapakov/events{/privacy}","received_events_url":"https://api.github.com/users/vshlapakov/received_events","type":"User","site_admin":false},"created_at":"2015-03-12T09:34:55Z","updated_at":"2015-03-12T09:34:55Z","body":"Done!"}}