{"action":"created","issue":{"url":"https://api.github.com/repos/mapnik/node-mapnik/issues/419","labels_url":"https://api.github.com/repos/mapnik/node-mapnik/issues/419/labels{/name}","comments_url":"https://api.github.com/repos/mapnik/node-mapnik/issues/419/comments","events_url":"https://api.github.com/repos/mapnik/node-mapnik/issues/419/events","html_url":"https://github.com/mapnik/node-mapnik/issues/419","id":62612160,"number":419,"title":"Stop storing `vector_tile::Tile` inside mapnik.VectorTile object","user":{"login":"springmeyer","id":20300,"avatar_url":"https://avatars.githubusercontent.com/u/20300?v=3","gravatar_id":"","url":"https://api.github.com/users/springmeyer","html_url":"https://github.com/springmeyer","followers_url":"https://api.github.com/users/springmeyer/followers","following_url":"https://api.github.com/users/springmeyer/following{/other_user}","gists_url":"https://api.github.com/users/springmeyer/gists{/gist_id}","starred_url":"https://api.github.com/users/springmeyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/springmeyer/subscriptions","organizations_url":"https://api.github.com/users/springmeyer/orgs","repos_url":"https://api.github.com/users/springmeyer/repos","events_url":"https://api.github.com/users/springmeyer/events{/privacy}","received_events_url":"https://api.github.com/users/springmeyer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-18T07:24:41Z","updated_at":"2015-03-23T22:37:54Z","closed_at":null,"body":"The `class VectorTile` inside node-mapnik stores both:\r\n\r\n```cpp\r\nstd::string buffer_;\r\n```\r\nand:\r\n\r\n```cpp\r\nvector_tile::Tile tiledata_;\r\n```\r\n\r\nThe `buffer_` represents a serialized protobuf string. The `tiledata_` represents in-memory protobuf objects. The `parsing_status` enum tries to manage state about whether the `buffer_` represents new data that needs to be parsed into the `tiledata_`. This requires the user to `vtile.parse()` in order for certain functions like `vtile.toJSON` or `vtile.toGeoJSON` to work. \r\n\r\nOverall this is horrible design and needs to go:\r\n\r\n  - managing state internally like this is very error prone\r\n  - memory is wasted by storing both the protobuf string and in-memory objects\r\n  - lots of switches in the code try to work off the `buffer_` if it looks like it represents all the data, but this is dubious\r\n\r\nWhy did this come about?\r\n\r\n  - Originally only the `tiledata_` was stored which was fine and clean\r\n  - But when we added `vtile.composite` the need for performance led to needing to concatenate buffers for the fastest return when z/x/y match. Because `SerializeToString` is slow we started storing the `buffer_` and if it is populated this made possible this optimization: https://github.com/mapnik/node-mapnik/blob/e37e1cdb543c54bef72db71a9ae6c2ef548a49fd/src/mapnik_vector_tile.cpp#L381-L385\r\n  - And then [this path](https://github.com/mapnik/node-mapnik/blob/e37e1cdb543c54bef72db71a9ae6c2ef548a49fd/src/mapnik_vector_tile.cpp#L386) is only taken if a vile is passed in that has not previously had `vtile.parse()` called\r\n\r\nThe way forward:\r\n\r\n  - Only store `buffer_` on the vtile. Whenever data is added to the vtile serialize it to this `buffer_` either replacing or appending.\r\n  - This way there will no longer be a hard dependency on the libprotobuf generated `vector_tile::Tile`. In the future a different protobuf encoder could be used.\r\n  -  Also then we can use any protobuf decoder to read the data from the current `buffer_`. Specifically we can move to pbf.hpp (which is already being used for `vtile.names()`) but should be used for more operations: https://github.com/mapbox/mapnik-vector-tile/issues/92\r\n\r\nThe final goal would be to be able to do:\r\n\r\n```js\r\nvtile.setData(<buffer>)\r\n// no vtile.parse() is needed anymore\r\nvtile.toGeoJSON() // internally pbf.hpp is used inside vector_tile_datasource so no `vector_tile::Tile` needs to be allocated\r\n```"},"comment":{"url":"https://api.github.com/repos/mapnik/node-mapnik/issues/comments/85234628","html_url":"https://github.com/mapnik/node-mapnik/issues/419#issuecomment-85234628","issue_url":"https://api.github.com/repos/mapnik/node-mapnik/issues/419","id":85234628,"user":{"login":"springmeyer","id":20300,"avatar_url":"https://avatars.githubusercontent.com/u/20300?v=3","gravatar_id":"","url":"https://api.github.com/users/springmeyer","html_url":"https://github.com/springmeyer","followers_url":"https://api.github.com/users/springmeyer/followers","following_url":"https://api.github.com/users/springmeyer/following{/other_user}","gists_url":"https://api.github.com/users/springmeyer/gists{/gist_id}","starred_url":"https://api.github.com/users/springmeyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/springmeyer/subscriptions","organizations_url":"https://api.github.com/users/springmeyer/orgs","repos_url":"https://api.github.com/users/springmeyer/repos","events_url":"https://api.github.com/users/springmeyer/events{/privacy}","received_events_url":"https://api.github.com/users/springmeyer/received_events","type":"User","site_admin":false},"created_at":"2015-03-23T22:37:54Z","updated_at":"2015-03-23T22:37:54Z","body":"When this is done these funky tests should start working and can be enabled again: https://github.com/mapnik/node-mapnik/blob/master/test/vector-tile.composite.test.js#L361-L391"}}