{"action":"created","issue":{"url":"https://api.github.com/repos/openaddresses/machine/issues/210","labels_url":"https://api.github.com/repos/openaddresses/machine/issues/210/labels{/name}","comments_url":"https://api.github.com/repos/openaddresses/machine/issues/210/comments","events_url":"https://api.github.com/repos/openaddresses/machine/issues/210/events","html_url":"https://github.com/openaddresses/machine/issues/210","id":105730780,"number":210,"title":"Update dot map on a schedule","user":{"login":"migurski","id":58730,"avatar_url":"https://avatars.githubusercontent.com/u/58730?v=3","gravatar_id":"","url":"https://api.github.com/users/migurski","html_url":"https://github.com/migurski","followers_url":"https://api.github.com/users/migurski/followers","following_url":"https://api.github.com/users/migurski/following{/other_user}","gists_url":"https://api.github.com/users/migurski/gists{/gist_id}","starred_url":"https://api.github.com/users/migurski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/migurski/subscriptions","organizations_url":"https://api.github.com/users/migurski/orgs","repos_url":"https://api.github.com/users/migurski/repos","events_url":"https://api.github.com/users/migurski/events{/privacy}","received_events_url":"https://api.github.com/users/migurski/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openaddresses/machine/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-09-10T04:46:12Z","updated_at":"2015-09-27T17:15:11Z","closed_at":null,"body":"Adapt [@sbma44’s `make_mbtiles.sh` script](https://github.com/openaddresses/openaddresses.io/blob/gh-pages/_src/make_mbtiles.sh) to a set of chef recipes and a thing we can run."},"comment":{"url":"https://api.github.com/repos/openaddresses/machine/issues/comments/143577919","html_url":"https://github.com/openaddresses/machine/issues/210#issuecomment-143577919","issue_url":"https://api.github.com/repos/openaddresses/machine/issues/210","id":143577919,"user":{"login":"ericfischer","id":1951835,"avatar_url":"https://avatars.githubusercontent.com/u/1951835?v=3","gravatar_id":"","url":"https://api.github.com/users/ericfischer","html_url":"https://github.com/ericfischer","followers_url":"https://api.github.com/users/ericfischer/followers","following_url":"https://api.github.com/users/ericfischer/following{/other_user}","gists_url":"https://api.github.com/users/ericfischer/gists{/gist_id}","starred_url":"https://api.github.com/users/ericfischer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericfischer/subscriptions","organizations_url":"https://api.github.com/users/ericfischer/orgs","repos_url":"https://api.github.com/users/ericfischer/repos","events_url":"https://api.github.com/users/ericfischer/events{/privacy}","received_events_url":"https://api.github.com/users/ericfischer/received_events","type":"User","site_admin":false},"created_at":"2015-09-27T17:15:11Z","updated_at":"2015-09-27T17:15:11Z","body":"The temporary files are in `/tmp` or wherever you specify with `-t`. They are deleted on creation so they are only a nameless inode most of the time, but you can see their original names in `/proc/*/fd` if you look.\r\n\r\nThe memory seems fairly believable. During parsing, the only thing that is in memory is the JSON parse tree itself; the rest is being streamed to disk.\r\n\r\nThe intensive part is immediately after that (\"Reordering geometry\"), where it maps the geometry file into memory and writes it out to a new temporary file ordered by quadtree index instead of the input order. This can thrash a lot because it is not sequential at all unless the original data was in close to that order. That's probably where you saw 8.5GB of memory used, and I should do a new release because it could be smaller than 46 bytes now.\r\n\r\nThen the tiling part is basically sequential: reading from temporary files and writing to new temporary files, and at each moment having only the features that will actually appear in the final output of a single tile in memory at any one moment. It is not a very efficient format in memory (64-bit drawing instructions, 64-bit coordinates) but I didn't try to crunch it because it's only a tile at a time.\r\n\r\nThe time seems fairly believable. Parsing GeoJSON is slow because JSON is complicated. I should make another input format that doesn't spend so much time on punctuation."}}