{"action":"created","issue":{"url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/351","labels_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/351/labels{/name}","comments_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/351/comments","events_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/351/events","html_url":"https://github.com/openstreetmap/osm2pgsql/issues/351","id":79551989,"number":351,"title":"OOM killed after processing phase","user":{"login":"pnorman","id":1190866,"avatar_url":"https://avatars.githubusercontent.com/u/1190866?v=3","gravatar_id":"","url":"https://api.github.com/users/pnorman","html_url":"https://github.com/pnorman","followers_url":"https://api.github.com/users/pnorman/followers","following_url":"https://api.github.com/users/pnorman/following{/other_user}","gists_url":"https://api.github.com/users/pnorman/gists{/gist_id}","starred_url":"https://api.github.com/users/pnorman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pnorman/subscriptions","organizations_url":"https://api.github.com/users/pnorman/orgs","repos_url":"https://api.github.com/users/pnorman/repos","events_url":"https://api.github.com/users/pnorman/events{/privacy}","received_events_url":"https://api.github.com/users/pnorman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/openstreetmap/osm2pgsql/milestones/4","html_url":"https://github.com/openstreetmap/osm2pgsql/milestones/0.88.0","labels_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/milestones/4/labels","id":1126710,"number":4,"title":"0.88.0","description":"Issues critical for the 0.88.0 release","creator":{"login":"pnorman","id":1190866,"avatar_url":"https://avatars.githubusercontent.com/u/1190866?v=3","gravatar_id":"","url":"https://api.github.com/users/pnorman","html_url":"https://github.com/pnorman","followers_url":"https://api.github.com/users/pnorman/followers","following_url":"https://api.github.com/users/pnorman/following{/other_user}","gists_url":"https://api.github.com/users/pnorman/gists{/gist_id}","starred_url":"https://api.github.com/users/pnorman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pnorman/subscriptions","organizations_url":"https://api.github.com/users/pnorman/orgs","repos_url":"https://api.github.com/users/pnorman/repos","events_url":"https://api.github.com/users/pnorman/events{/privacy}","received_events_url":"https://api.github.com/users/pnorman/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":0,"state":"open","created_at":"2015-05-23T03:20:47Z","updated_at":"2015-05-25T20:24:31Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2015-05-22T18:06:08Z","updated_at":"2015-07-01T04:36:25Z","closed_at":null,"body":"On a machine with 32GB RAM, osm2pgsql 0.87.3, command line `~/osm2pgsql/osm2pgsql -d gis --cache 26000 --flat-nodes nodes.bin --slim --drop --\r\nnumber-processes 8 -S ~/openstreetmap-carto/openstreetmap-carto.style planet-150202.osm.pbf`\r\n\r\n```\r\nosm2pgsql SVN version 0.87.3 (64bit id space)\r\n\r\nUsing built-in tag processing pipeline\r\nUsing projection SRS 900913 (Spherical Mercator)\r\nSetting up table: planet_osm_point\r\nSetting up table: planet_osm_line\r\nSetting up table: planet_osm_polygon\r\nSetting up table: planet_osm_roads\r\nAllocating memory for dense node cache\r\nAllocating dense node cache in one big chunk\r\nAllocating memory for sparse node cache\r\nSharing dense sparse\r\nNode-cache: cache=26000MB, maxblocks=3328000*8192, allocation method=11\r\nMid: loading persistent node cache from nodes.bin\r\nAllocated space for persistent node cache file\r\nMaximum node in persistent node cache: 0\r\nMid: pgsql, scale=100 cache=26000\r\nSetting up table: planet_osm_nodes\r\nSetting up table: planet_osm_ways\r\nSetting up table: planet_osm_rels\r\n\r\nReading in file: planet-150202.osm.pbf\r\nProcessing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2630170 269.21/s)\r\nStandard exception processing relation id=3890452: TopologyException: side location conflict at 1038877.48 5779728.6200000001\r\nProcessing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2841950 273.71/s)\r\nStandard exception processing relation id=4136486: TopologyException: side location conflict at 2924456.4100000001 9012036.7799999993\r\nProcessing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2881850 275.04/s)\r\nStandard exception processing relation id=4180656: TopologyException: side location conflict at 272703.48999999999 5093940.6600000001\r\nProcessing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(3100780 281.58/s)\r\nStandard exception processing relation id=4542207: TopologyException: side location conflict at 15042893.51 4046892.75\r\nProcessing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(3109640 282.05/s)  parse time: 17314s\r\nNode stats: total(2715748725), max(3326469484) in 1046s\r\nWay stats: total(272021914), max(326028236) in 5243s\r\nRelation stats: total(3109645), max(4551629) in 11025s\r\nMaximum node in persistent node cache: 3327131647\r\nCommitting transaction for planet_osm_point\r\nCommitting transaction for planet_osm_line\r\nCommitting transaction for planet_osm_polygon\r\nCommitting transaction for planet_osm_roads\r\nCommand terminated by signal 9\r\n```\r\n\r\nWatching `status` in /proc/PID tells me that about 24500 cache would have been sufficient.\r\n\r\nThere are about 8GB free as it finishes processing\r\n\r\n![image](https://cloud.githubusercontent.com/assets/1190866/7776708/de58f4b2-0071-11e5-8566-602376c3f0f1.png)\r\n\r\nI haven't yet looked through the code to see what could be using large amounts of memory. `vm.overcommit_memory` is set to 1, so an allocation alone won't cause a problem, only use. `vm.overcommit_ratio` is 50, but I don't believe that matters with overcommit_memory set to 1."},"comment":{"url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/comments/117432368","html_url":"https://github.com/openstreetmap/osm2pgsql/issues/351#issuecomment-117432368","issue_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/351","id":117432368,"user":{"login":"pnorman","id":1190866,"avatar_url":"https://avatars.githubusercontent.com/u/1190866?v=3","gravatar_id":"","url":"https://api.github.com/users/pnorman","html_url":"https://github.com/pnorman","followers_url":"https://api.github.com/users/pnorman/followers","following_url":"https://api.github.com/users/pnorman/following{/other_user}","gists_url":"https://api.github.com/users/pnorman/gists{/gist_id}","starred_url":"https://api.github.com/users/pnorman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pnorman/subscriptions","organizations_url":"https://api.github.com/users/pnorman/orgs","repos_url":"https://api.github.com/users/pnorman/repos","events_url":"https://api.github.com/users/pnorman/events{/privacy}","received_events_url":"https://api.github.com/users/pnorman/received_events","type":"User","site_admin":false},"created_at":"2015-07-01T04:36:25Z","updated_at":"2015-07-01T04:36:25Z","body":"I tried --without-lockfree, and the RAM usage is substantially better\r\n\r\n![image](https://cloud.githubusercontent.com/assets/1190866/8447581/d7f7cb8a-1f6f-11e5-9669-3694f6876498.png)\r\n\r\nSo, I guess the options are\r\n\r\n1. Fix the boost::lockfree::queue implementation to use less memory\r\n2. Switch the configure logic to default to --without-lockfree\r\n3. Drop the boost::lockfree::queue implementation and simplify ./configure"}}