{"action":"created","issue":{"url":"https://api.github.com/repos/mapnik/mapnik-support/issues/57","labels_url":"https://api.github.com/repos/mapnik/mapnik-support/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/mapnik/mapnik-support/issues/57/comments","events_url":"https://api.github.com/repos/mapnik/mapnik-support/issues/57/events","html_url":"https://github.com/mapnik/mapnik-support/issues/57","id":123100419,"number":57,"title":"threaded mapnik with multiple layers lead to missing/uncompleted layers in random tiles","user":{"login":"KaNe23","id":241655,"avatar_url":"https://avatars.githubusercontent.com/u/241655?v=3","gravatar_id":"","url":"https://api.github.com/users/KaNe23","html_url":"https://github.com/KaNe23","followers_url":"https://api.github.com/users/KaNe23/followers","following_url":"https://api.github.com/users/KaNe23/following{/other_user}","gists_url":"https://api.github.com/users/KaNe23/gists{/gist_id}","starred_url":"https://api.github.com/users/KaNe23/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KaNe23/subscriptions","organizations_url":"https://api.github.com/users/KaNe23/orgs","repos_url":"https://api.github.com/users/KaNe23/repos","events_url":"https://api.github.com/users/KaNe23/events{/privacy}","received_events_url":"https://api.github.com/users/KaNe23/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-19T17:40:39Z","updated_at":"2015-12-20T22:06:12Z","closed_at":null,"body":"Hello, I'm using mapnik to generate tiles for a leaflet powered web-application map.\r\n\r\nThe generated tiles are administrative boarders of Rwanda. To display a deeper administrative \r\nlayer when zooming into the map, i split the geodate into different layers and display them according to the maxzoom.\r\n\r\nAs tiles-servers i tried the tilelite server (https://github.com/springmeyer/tilelite) and tilecache (http://tilecache.org/).\r\n\r\nBoth providing the functionality to render the tiles threaded.\r\n\r\nNow i run into the problem, that if i activated threading, that the layers defined further below in the \r\nmapnik xml file started to be incomplete or missing in random tiles. This happened regardless of the data source. I tried both, a postgis database and the corresponding shape files. I also tested the \r\npostgis queries in qgis and they looked fine.\r\n\r\nIt was easier to reproduce the behaviour with the postgis database and after shuffling the layers around in the xml file, it were always the layers further below missing partly. Deactivating the cache\r\nand requesting the broken tiles again in a single browser tab was resulting in a perfect fine tile.\r\nIt has something to do with requesting lots of tiles parallel with threads.\r\n\r\nHere is a sample:\r\n![incompleted_tiles](https://cloud.githubusercontent.com/assets/241655/11914329/fa6ab520-a67d-11e5-9c76-6b7c34cf54a6.png)\r\n\r\nIt should basically look like this:\r\n![reference](https://cloud.githubusercontent.com/assets/241655/11914366/8970edf6-a67f-11e5-86f2-14c545651274.png)\r\n\r\nDeactivating the threading in both cases fixed the issue, but obviously made the whole process slower.\r\n\r\nHere is my mapnik xml file:\r\n\r\n```xml\r\n<Map srs=\"+init=epsg:3857\">\r\n\r\n  <Style name=\"provStyle\" filter-mode=\"first\">\r\n    <Rule>\r\n      <MaxScaleDenominator>280000000</MaxScaleDenominator>\r\n      <MinScaleDenominator>4367830</MinScaleDenominator>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"purple\" stroke-width=\"2\" stroke-opacity=\"0.5\" stroke-linejoin=\"round\"/>\r\n    </Rule>\r\n    <Rule>\r\n      <MaxScaleDenominator>4367830</MaxScaleDenominator>\r\n      <MinScaleDenominator>0</MinScaleDenominator>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"purple\" stroke-width=\"4\" stroke-opacity=\"0.5\" stroke-linejoin=\"round\"/>\r\n    </Rule>\r\n  </Style>\r\n\r\n  <Style name=\"distStyle\" filter-mode=\"first\">\r\n    <Rule>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"black\" stroke-width=\"1\" stroke-opacity=\"0.7\" stroke-linejoin=\"round\" />\r\n    </Rule>\r\n  </Style>\r\n\r\n  <Style name=\"sectStyle\" filter-mode=\"first\">\r\n    <Rule>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"black\" stroke-width=\"0.6\" stroke-opacity=\"0.75\" />\r\n    </Rule>\r\n  </Style>\r\n\r\n  <Style name=\"cellStyle\" filter-mode=\"first\">\r\n    <Rule>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"black\" stroke-width=\"0.5\" stroke-opacity=\"0.75\" />\r\n    </Rule>\r\n  </Style>\r\n\r\n  <Style name=\"villStyle\" filter-mode=\"first\">\r\n    <Rule>\r\n      <PolygonSymbolizer fill=\"black\" fill-opacity=\"0.0\"/>\r\n      <LineSymbolizer stroke=\"black\" stroke-width=\"0.3\" stroke-opacity=\"0.75\" />\r\n    </Rule>\r\n  </Style>\r\n\r\n  <Layer name=\"prov\" srs=\"+init=epsg:4326\" >\r\n    <StyleName>provStyle</StyleName>\r\n    <Datasource>\r\n      <Parameter name=\"type\">postgis</Parameter>\r\n      <Parameter name=\"host\">localhost</Parameter>\r\n      <Parameter name=\"dbname\">postgres</Parameter>\r\n      <Parameter name=\"user\">postgres</Parameter>\r\n      <Parameter name=\"password\">root</Parameter>\r\n      <Parameter name=\"table\">rwanda_prov</Parameter>\r\n      <!-- <Parameter name=\"type\">shape</Parameter>\r\n      <Parameter name=\"file\">rwanda_prov.shp</Parameter> -->\r\n    </Datasource>\r\n  </Layer>\r\n\r\n  <Layer name=\"dist\" srs=\"+init=epsg:4326\" maxzoom=\"4368000\">\r\n    <StyleName>distStyle</StyleName>\r\n    <Datasource>\r\n      <Parameter name=\"type\">postgis</Parameter>\r\n      <Parameter name=\"host\">localhost</Parameter>\r\n      <Parameter name=\"dbname\">postgres</Parameter>\r\n      <Parameter name=\"user\">postgres</Parameter>\r\n      <Parameter name=\"password\">root</Parameter>\r\n      <Parameter name=\"table\">rwanda_dist</Parameter>\r\n      <!-- <Parameter name=\"type\">shape</Parameter>\r\n      <Parameter name=\"file\">rwanda_dist.shp</Parameter> -->\r\n    </Datasource>\r\n  </Layer>\r\n\r\n  <Layer name=\"sect\" srs=\"+init=epsg:4326\" maxzoom=\"1092000\" >\r\n    <StyleName>sectStyle</StyleName>\r\n    <Datasource>\r\n      <Parameter name=\"type\">postgis</Parameter>\r\n      <Parameter name=\"host\">localhost</Parameter>\r\n      <Parameter name=\"dbname\">postgres</Parameter>\r\n      <Parameter name=\"user\">postgres</Parameter>\r\n      <Parameter name=\"password\">root</Parameter>\r\n      <Parameter name=\"table\">rwanda_sect</Parameter>\r\n      <!-- <Parameter name=\"type\">shape</Parameter>\r\n      <Parameter name=\"file\">rwanda_sect.shp</Parameter> -->\r\n    </Datasource>\r\n  </Layer>\r\n\r\n  <Layer name=\"cell\" srs=\"+init=epsg:4326\" maxzoom=\"546000\" >\r\n    <StyleName>cellStyle</StyleName>\r\n    <Datasource>\r\n      <Parameter name=\"type\">postgis</Parameter>\r\n      <Parameter name=\"host\">localhost</Parameter>\r\n      <Parameter name=\"dbname\">postgres</Parameter>\r\n      <Parameter name=\"user\">postgres</Parameter>\r\n      <Parameter name=\"password\">root</Parameter>\r\n      <Parameter name=\"table\">rwanda_cell</Parameter>\r\n      <!-- <Parameter name=\"type\">shape</Parameter>\r\n      <Parameter name=\"file\">rwanda_cell.shp</Parameter> -->\r\n    </Datasource>\r\n  </Layer>\r\n\r\n  <Layer name=\"vill\" srs=\"+init=epsg:4326\" maxzoom=\"68300\" >\r\n    <StyleName>villStyle</StyleName>\r\n    <Datasource>\r\n      <Parameter name=\"type\">postgis</Parameter>\r\n      <Parameter name=\"host\">localhost</Parameter>\r\n      <Parameter name=\"dbname\">postgres</Parameter>\r\n      <Parameter name=\"user\">postgres</Parameter>\r\n      <Parameter name=\"password\">root</Parameter>\r\n      <Parameter name=\"table\">rwanda_vill</Parameter>\r\n      <!-- <Parameter name=\"type\">shape</Parameter>\r\n      <Parameter name=\"file\">rwanda_vill.shp</Parameter> -->\r\n    </Datasource>\r\n  </Layer>\r\n</Map>\r\n```\r\n\r\nHas anyone some ideas, if i do something wrong or is this a mapnik issue?\r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/mapnik/mapnik-support/issues/comments/166158143","html_url":"https://github.com/mapnik/mapnik-support/issues/57#issuecomment-166158143","issue_url":"https://api.github.com/repos/mapnik/mapnik-support/issues/57","id":166158143,"user":{"login":"springmeyer","id":20300,"avatar_url":"https://avatars.githubusercontent.com/u/20300?v=3","gravatar_id":"","url":"https://api.github.com/users/springmeyer","html_url":"https://github.com/springmeyer","followers_url":"https://api.github.com/users/springmeyer/followers","following_url":"https://api.github.com/users/springmeyer/following{/other_user}","gists_url":"https://api.github.com/users/springmeyer/gists{/gist_id}","starred_url":"https://api.github.com/users/springmeyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/springmeyer/subscriptions","organizations_url":"https://api.github.com/users/springmeyer/orgs","repos_url":"https://api.github.com/users/springmeyer/repos","events_url":"https://api.github.com/users/springmeyer/events{/privacy}","received_events_url":"https://api.github.com/users/springmeyer/received_events","type":"User","site_admin":false},"created_at":"2015-12-20T22:06:11Z","updated_at":"2015-12-20T22:06:11Z","body":"Those two server implementations only support multiprocess usage. Multithreaded usage is not supported (for example see https://github.com/springmeyer/tilelite/blob/master/tilelite.wsgi#L21-L22). This is exactly for the reason you see: `mapnik.Map` objects are, by design, insteaded to be only used within one thread at a time and therefore you should not use the same one across multiple threads. If you deploy those servers in threaded mode then the maps are being shared across threads which is invalid usage.\r\n\r\nThat said, yes multithreading is optimal for best performance and efficient memory usage. So how do you render maps with Mapnik in multiple threads then? Well, simple: you pool maps such that you can:\r\n\r\n 1) Use and re-use available `mapnik.Map` objects in rendering threads\r\n 2) No one `mapnik.Map` object will be in use in more than one thread at any time\r\n\r\nAll production ready map server implementations that I know of either 1) run multiprocess only, or 2) pool maps to enable safe and fast multithreading. There is likely a python server out there that does pooling but I'm not aware of one myself. I maintain a Node.js Mapnik stack of tools that do pooling, see https://github.com/mapnik/node-mapnik-sample-code/blob/master/tile/pool/app.js for an example and https://github.com/mapbox/tilelive-mapnik for a more fully fledged implementation."}}