{"action":"created","issue":{"url":"https://api.github.com/repos/mapbox/tile-reduce/issues/46","labels_url":"https://api.github.com/repos/mapbox/tile-reduce/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/mapbox/tile-reduce/issues/46/comments","events_url":"https://api.github.com/repos/mapbox/tile-reduce/issues/46/events","html_url":"https://github.com/mapbox/tile-reduce/pull/46","id":114264593,"number":46,"title":"Next generation Tile Reduce","user":{"login":"mourner","id":25395,"avatar_url":"https://avatars.githubusercontent.com/u/25395?v=3","gravatar_id":"","url":"https://api.github.com/users/mourner","html_url":"https://github.com/mourner","followers_url":"https://api.github.com/users/mourner/followers","following_url":"https://api.github.com/users/mourner/following{/other_user}","gists_url":"https://api.github.com/users/mourner/gists{/gist_id}","starred_url":"https://api.github.com/users/mourner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mourner/subscriptions","organizations_url":"https://api.github.com/users/mourner/orgs","repos_url":"https://api.github.com/users/mourner/repos","events_url":"https://api.github.com/users/mourner/events{/privacy}","received_events_url":"https://api.github.com/users/mourner/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mapbox/tile-reduce/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":35,"created_at":"2015-10-30T12:59:38Z","updated_at":"2015-11-20T16:49:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mapbox/tile-reduce/pulls/46","html_url":"https://github.com/mapbox/tile-reduce/pull/46","diff_url":"https://github.com/mapbox/tile-reduce/pull/46.diff","patch_url":"https://github.com/mapbox/tile-reduce/pull/46.patch"},"body":"An early version of the next-generation Tile Reduce that is **thousands of times faster** than the current one. This is _not_ a joke. \r\n\r\nFor example, generating GeoJSON road difference between OSM & Tiger 2015 for US West z12 (~26% of US) takes **2 minutes 45 seconds** on my Macbook Pro; previously it would take **a day** on a much more powerful EC2 c3.4xlarge instance eating all of the available RAM. \r\n\r\nThe new architecture also makes Tile Reduce much more flexible and loosely coupled, e.g. easy to extend for raster support, weird tile formats, lazy decoding, etc. \r\n\r\nI'll devlog how I did this, but the main idea behind the rewrite is removing almost all communication between the main thread and the workers and avoiding unnecessary work:\r\n\r\n- responsibility for fetching/parsing tiles fully moves to the workers\r\n- vector tiles are not parsed and converted to GeoJSON by default; instead, VT layer objects are passed to the workers for lazy decoding (parsing only the features that are relevant to the current task)\r\n- big GeoJSON output like difference is never passed back to the main thread; instead, it's streamed directly from workers to stdout; `reduce` event is only for smaller values like statistics\r\n\r\nSomewhat breaks API compatibility, but this is something we have to do to support the new architecture. Upgrading existing processors should be easy once we finalize the work.\r\n\r\nAlso sports a fancy progress bar so that you could _feel_ the unreal performance:\r\n\r\n![tilereduce-progress](https://cloud.githubusercontent.com/assets/25395/10845973/b7cfc43e-7f14-11e5-9e6c-84d42d997cab.gif)\r\n\r\nWhile all the basic stuff works, this is still a work in progress. Rough list of remaining tasks:\r\n\r\n- [x] clean up and push the new diff processor example (today)\r\n- [x] get feedback on the API changes\r\n- [x] code review by the team\r\n- [x] readd remote tiles support\r\n- [x] readd non-bbox covers\r\n- [x] streaming tile list input\r\n- [x] avoid overloading the message queue\r\n- [x] ~~readd overzoom (slicing to higher zoom levels than the sources provide)~~\r\n- [x] fix interleaved output issue @tcql\r\n- [x] readd tests (in progress) @morganherlocker \r\n- [x] readd rate limiting\r\n- [x] support streaming tile list from mbtiles directly @mourner\r\n- [x] documentation & upgrade guide\r\n\r\nI would love feedback and help on this. We could win so much with a blazing fast Tile Reduce, so let's make it happen sooner.\r\n\r\ncc @morganherlocker @MateoV @aaronlidman @tcql @tmcw @lxbarth @ericfischer @batpad @Rub21 @geohacker @arunasank @isiyu "},"comment":{"url":"https://api.github.com/repos/mapbox/tile-reduce/issues/comments/158457173","html_url":"https://github.com/mapbox/tile-reduce/pull/46#issuecomment-158457173","issue_url":"https://api.github.com/repos/mapbox/tile-reduce/issues/46","id":158457173,"user":{"login":"tcql","id":348805,"avatar_url":"https://avatars.githubusercontent.com/u/348805?v=3","gravatar_id":"","url":"https://api.github.com/users/tcql","html_url":"https://github.com/tcql","followers_url":"https://api.github.com/users/tcql/followers","following_url":"https://api.github.com/users/tcql/following{/other_user}","gists_url":"https://api.github.com/users/tcql/gists{/gist_id}","starred_url":"https://api.github.com/users/tcql/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tcql/subscriptions","organizations_url":"https://api.github.com/users/tcql/orgs","repos_url":"https://api.github.com/users/tcql/repos","events_url":"https://api.github.com/users/tcql/events{/privacy}","received_events_url":"https://api.github.com/users/tcql/received_events","type":"User","site_admin":false},"created_at":"2015-11-20T16:49:40Z","updated_at":"2015-11-20T16:49:40Z","body":"working on adding a short doc section about events, so we've got bases covered"}}