{"action":"created","issue":{"url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/17","labels_url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/17/comments","events_url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/17/events","html_url":"https://github.com/geo-data/cesium-terrain-builder/issues/17","id":60017895,"number":17,"title":"Threading not working well for ctb-tile?","user":{"login":"GeorgeDewar","id":5344417,"avatar_url":"https://avatars.githubusercontent.com/u/5344417?v=3","gravatar_id":"","url":"https://api.github.com/users/GeorgeDewar","html_url":"https://github.com/GeorgeDewar","followers_url":"https://api.github.com/users/GeorgeDewar/followers","following_url":"https://api.github.com/users/GeorgeDewar/following{/other_user}","gists_url":"https://api.github.com/users/GeorgeDewar/gists{/gist_id}","starred_url":"https://api.github.com/users/GeorgeDewar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GeorgeDewar/subscriptions","organizations_url":"https://api.github.com/users/GeorgeDewar/orgs","repos_url":"https://api.github.com/users/GeorgeDewar/repos","events_url":"https://api.github.com/users/GeorgeDewar/events{/privacy}","received_events_url":"https://api.github.com/users/GeorgeDewar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-05T21:20:35Z","updated_at":"2015-03-08T23:03:25Z","closed_at":null,"body":"We are getting approx 150 tiles per second processed on a layer 16 run with ctb-tile if we specify 1 thread, rising only to 400 tiles per second at 8 threads (on an 8-core machine).\r\n\r\nWe have tried this on a 36-core machine and the performance with 36 threads (the default setting being 1 per core) is no improvement whatsoever over 8 threads.\r\n\r\nThese are both EC2 servers. We have tried using a RAM drive for storage to eliminate the possibly of a disk bottleneck. We tend to see a load average of around the number of threads, and CPU utilization per core of around 70%.\r\n\r\nIf we ran another job at the same time, we can get utilization to 100%, but overall throughput is actually less.\r\n\r\nThe most peculiar thing is that on an Intel Core i5 destop machine, we get about 2000 tiles per second - dramatically better performance than the 36-core AWS box. We are not sure why.\r\n\r\nAny tips? Here's the command we're running:\r\n\r\n```ctb-tile --output-dir tiles3 -s 16 -e 16 -m 8000000000 -c n dem_wgs84.vrt```\r\n\r\nWhere n above is of course the various thread counts we have tried. Using more threads than cores also does not help, though I wouldn't expect it to.\r\n\r\nPlease let me know if any additional information would be helpful."},"comment":{"url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/comments/77780698","html_url":"https://github.com/geo-data/cesium-terrain-builder/issues/17#issuecomment-77780698","issue_url":"https://api.github.com/repos/geo-data/cesium-terrain-builder/issues/17","id":77780698,"user":{"login":"GeorgeDewar","id":5344417,"avatar_url":"https://avatars.githubusercontent.com/u/5344417?v=3","gravatar_id":"","url":"https://api.github.com/users/GeorgeDewar","html_url":"https://github.com/GeorgeDewar","followers_url":"https://api.github.com/users/GeorgeDewar/followers","following_url":"https://api.github.com/users/GeorgeDewar/following{/other_user}","gists_url":"https://api.github.com/users/GeorgeDewar/gists{/gist_id}","starred_url":"https://api.github.com/users/GeorgeDewar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GeorgeDewar/subscriptions","organizations_url":"https://api.github.com/users/GeorgeDewar/orgs","repos_url":"https://api.github.com/users/GeorgeDewar/repos","events_url":"https://api.github.com/users/GeorgeDewar/events{/privacy}","received_events_url":"https://api.github.com/users/GeorgeDewar/received_events","type":"User","site_admin":false},"created_at":"2015-03-08T23:03:25Z","updated_at":"2015-03-08T23:03:25Z","body":"Since writing this I have read some things that make me suspect that the bottleneck is the file system - not disk I/O, but creating files. Interesting..."}}