{"action":"created","issue":{"url":"https://api.github.com/repos/geoadmin/3d-forge/issues/104","labels_url":"https://api.github.com/repos/geoadmin/3d-forge/issues/104/labels{/name}","comments_url":"https://api.github.com/repos/geoadmin/3d-forge/issues/104/comments","events_url":"https://api.github.com/repos/geoadmin/3d-forge/issues/104/events","html_url":"https://github.com/geoadmin/3d-forge/pull/104","id":98299694,"number":104,"title":"TilerManger refactoring","user":{"login":"loicgasser","id":3306154,"avatar_url":"https://avatars.githubusercontent.com/u/3306154?v=3","gravatar_id":"","url":"https://api.github.com/users/loicgasser","html_url":"https://github.com/loicgasser","followers_url":"https://api.github.com/users/loicgasser/followers","following_url":"https://api.github.com/users/loicgasser/following{/other_user}","gists_url":"https://api.github.com/users/loicgasser/gists{/gist_id}","starred_url":"https://api.github.com/users/loicgasser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/loicgasser/subscriptions","organizations_url":"https://api.github.com/users/loicgasser/orgs","repos_url":"https://api.github.com/users/loicgasser/repos","events_url":"https://api.github.com/users/loicgasser/events{/privacy}","received_events_url":"https://api.github.com/users/loicgasser/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-07-31T01:51:51Z","updated_at":"2015-07-31T09:29:30Z","closed_at":"2015-07-31T09:13:52Z","pull_request":{"url":"https://api.github.com/repos/geoadmin/3d-forge/pulls/104","html_url":"https://github.com/geoadmin/3d-forge/pull/104","diff_url":"https://github.com/geoadmin/3d-forge/pull/104.diff","patch_url":"https://github.com/geoadmin/3d-forge/pull/104.patch"},"body":"This PR is a cleanup of the actual multiprocessing implementation.\r\n\r\nNow with @gjn concern regarding: https://github.com/geoadmin/3d-forge/issues/102\r\n\r\nThe way pool.map_async works is that it will actually consumes the entire generator and will then dispatch the jobs to the different processes.\r\n\r\nAs a result, each process ends up with a predefined set of jobs and all the jobs are hold in the memory.\r\n\r\nNow, all tiles are not equal and some need more processing time than others. What you call degrading is I think only the result of the Pool implementation of map_async.\r\n\r\nYou can actually reproduce this effect at a smaller scale if you decide to only generate level 12 of the TMS pyramid for instance.\r\n\r\nI found this reference on stackoverflow: http://stackoverflow.com/questions/5318936/python-multiprocessing-pool-lazy-iteration\r\n\r\nI believe that his idea of using itertools is not too bad but I don't think it is ideal in our case.\r\nI would rather implement a queing system that would dispatch a new job to the process as soon as the job is finished. This is the only way we can ensure that:\r\n\r\n1. jobs are generated thanks to a lazy iteration\r\n2. all the processes are always busy until the end\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/geoadmin/3d-forge/issues/comments/126621591","html_url":"https://github.com/geoadmin/3d-forge/pull/104#issuecomment-126621591","issue_url":"https://api.github.com/repos/geoadmin/3d-forge/issues/104","id":126621591,"user":{"login":"loicgasser","id":3306154,"avatar_url":"https://avatars.githubusercontent.com/u/3306154?v=3","gravatar_id":"","url":"https://api.github.com/users/loicgasser","html_url":"https://github.com/loicgasser","followers_url":"https://api.github.com/users/loicgasser/followers","following_url":"https://api.github.com/users/loicgasser/following{/other_user}","gists_url":"https://api.github.com/users/loicgasser/gists{/gist_id}","starred_url":"https://api.github.com/users/loicgasser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/loicgasser/subscriptions","organizations_url":"https://api.github.com/users/loicgasser/orgs","repos_url":"https://api.github.com/users/loicgasser/repos","events_url":"https://api.github.com/users/loicgasser/events{/privacy}","received_events_url":"https://api.github.com/users/loicgasser/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T09:29:30Z","updated_at":"2015-07-31T09:29:30Z","body":"A faster way to fix this problem would be to randomize the jobs within the list. It would probably greatly improve performance using 1 line of code."}}