{"action":"created","issue":{"url":"https://api.github.com/repos/pelias/openstreetmap/issues/82","labels_url":"https://api.github.com/repos/pelias/openstreetmap/issues/82/labels{/name}","comments_url":"https://api.github.com/repos/pelias/openstreetmap/issues/82/comments","events_url":"https://api.github.com/repos/pelias/openstreetmap/issues/82/events","html_url":"https://github.com/pelias/openstreetmap/pull/82","id":111337039,"number":82,"title":"Use Address-deduplicator","user":{"login":"hannesj","id":698100,"avatar_url":"https://avatars.githubusercontent.com/u/698100?v=3","gravatar_id":"","url":"https://api.github.com/users/hannesj","html_url":"https://github.com/hannesj","followers_url":"https://api.github.com/users/hannesj/followers","following_url":"https://api.github.com/users/hannesj/following{/other_user}","gists_url":"https://api.github.com/users/hannesj/gists{/gist_id}","starred_url":"https://api.github.com/users/hannesj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hannesj/subscriptions","organizations_url":"https://api.github.com/users/hannesj/orgs","repos_url":"https://api.github.com/users/hannesj/repos","events_url":"https://api.github.com/users/hannesj/events{/privacy}","received_events_url":"https://api.github.com/users/hannesj/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pelias/openstreetmap/labels/in%20review","name":"in review","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"orangejulius","id":111716,"avatar_url":"https://avatars.githubusercontent.com/u/111716?v=3","gravatar_id":"","url":"https://api.github.com/users/orangejulius","html_url":"https://github.com/orangejulius","followers_url":"https://api.github.com/users/orangejulius/followers","following_url":"https://api.github.com/users/orangejulius/following{/other_user}","gists_url":"https://api.github.com/users/orangejulius/gists{/gist_id}","starred_url":"https://api.github.com/users/orangejulius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orangejulius/subscriptions","organizations_url":"https://api.github.com/users/orangejulius/orgs","repos_url":"https://api.github.com/users/orangejulius/repos","events_url":"https://api.github.com/users/orangejulius/events{/privacy}","received_events_url":"https://api.github.com/users/orangejulius/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-10-14T06:57:31Z","updated_at":"2015-10-29T19:56:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pelias/openstreetmap/pulls/82","html_url":"https://github.com/pelias/openstreetmap/pull/82","diff_url":"https://github.com/pelias/openstreetmap/pull/82.diff","patch_url":"https://github.com/pelias/openstreetmap/pull/82.patch"},"body":"Hi,\r\n\r\nCurrently there is no possibility of using the address-deduplicator on openstreetmap addresses, as you can do with openaddresses. This adds the possibility of using pelias/address-deduplicator in the input stream to filter out already-seen addresses."},"comment":{"url":"https://api.github.com/repos/pelias/openstreetmap/issues/comments/152301291","html_url":"https://github.com/pelias/openstreetmap/pull/82#issuecomment-152301291","issue_url":"https://api.github.com/repos/pelias/openstreetmap/issues/82","id":152301291,"user":{"login":"orangejulius","id":111716,"avatar_url":"https://avatars.githubusercontent.com/u/111716?v=3","gravatar_id":"","url":"https://api.github.com/users/orangejulius","html_url":"https://github.com/orangejulius","followers_url":"https://api.github.com/users/orangejulius/followers","following_url":"https://api.github.com/users/orangejulius/following{/other_user}","gists_url":"https://api.github.com/users/orangejulius/gists{/gist_id}","starred_url":"https://api.github.com/users/orangejulius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orangejulius/subscriptions","organizations_url":"https://api.github.com/users/orangejulius/orgs","repos_url":"https://api.github.com/users/orangejulius/repos","events_url":"https://api.github.com/users/orangejulius/events{/privacy}","received_events_url":"https://api.github.com/users/orangejulius/received_events","type":"User","site_admin":false},"created_at":"2015-10-29T19:56:11Z","updated_at":"2015-10-29T19:56:11Z","body":"Hey @hannesj,\r\nThanks for adding this, it's a great additional option to have, and the config flag was a nice touch.\r\n\r\nI have a question though: how often do you see dupes in OSM data? I took a look at our Boston metroextract as an example. Without deduplication there were 14626 items imported, with deduplication there were 12091.\r\n\r\nHowever, a lot of the nodes and ways removed by the deduplicator are not real duplicates. It turns out the address deduplicator defaults to a [geohash precision of 5](https://github.com/openvenues/address_deduper/blob/master/address_deduper/config.py#L52), which ends up corresponding to about a 15km square due to the way the deduper works.\r\n\r\nThis is pretty big so in Boston for example, a lot of subway stops and chain stores (especially Dunkin' Donuts, which is very popular in Boston) are removed.\r\n\r\nAre there other places where OSM data isn't so good and a deduplicator is needed? Or are you telling the address deduplicator to use a higher precision? Very curious as it would help us decide how/if to roll this out in our Pelias instances.\r\n\r\n--------\r\nI did a bunch of analysis for this actually, so for you or anyone else I'll post my method here.\r\n\r\n1. Import the data with or without the deduplicator config settting (and a previously empty Elasticsearch index)\r\n2. dump all the data as JSON with this query (probably will break if run against a much larger dataset)\r\n\r\n    ```bash\r\n    curl -XPOST 'localhost:9200/pelias/_search' -d '\r\n    {\r\n      \"query\": {\r\n         \"match_all\": {}\r\n       },\r\n       \"size\": 20000,\r\n       \"sort\": [\"name.default\"]\r\n    }' > duplicated-es.json # change this to be correct for which setting you used\r\n    ```\r\n\r\n3. repeat for the other config setting\r\n4. parse the json into an array with only the name and lat/lon of each object using a script like this:\r\n\r\n    ```javascript\r\n    var fs = require('fs');\r\n    var file = fs.readFileSync(process.argv[2]);\r\n    var json = JSON.parse(file);\r\n\r\n    var names = json.hits.hits.map(function(hit) {\r\n      hit = hit['_source'];\r\n      return hit.name.default + \" \" + hit.center_point.lat + \",\" + hit.center_point.lon;\r\n    }).sort();\r\n\r\n    console.log(names);\r\n    ```\r\n5. diff the two outputs of that script and you'll get something like this\r\n\r\n    ```diff\r\n    --- duplicated.txt      2015-10-29 15:12:21.764554137 -0400\r\n    +++ deduplicated.txt    2015-10-29 15:14:55.880314347 -0400\r\n    @@ -1,5 +1,4 @@\r\n     [ '0 Charles Street 42.354356,-71.068275',\r\n    -  '0 New Washington Street 42.380042,-71.082336',\r\n       '1 42.358237,-71.092337',\r\n       '1 Adamson Street 42.358451,-71.136925',\r\n       '1 Akron Street 42.325887,-71.086411',\r\n    @@ -7,15 +6,10 @@\r\n       '1 Austin Park 42.367092,-71.104063',\r\n       '1 Avenue De Lafayette 42.353404,-71.060771',\r\n       '1 Batterymarch Park 42.229922,-71.02562',\r\n    -  '1 Batterymarch Park 42.229922,-71.02562',\r\n       '1 Beacon Street 42.358394,-71.060756',\r\n       '1 Belgrade Avenue 42.287487,-71.130317',\r\n       '1 Belmont Street 42.374856,-71.150301',\r\n    -  '1 Broadway 42.362634,-71.083658',\r\n    -  '1 Broadway 42.362665,-71.083477',\r\n       '1 Broadway 42.362744,-71.083786',\r\n    -  '1 Broadway 42.362771,-71.083278',\r\n    ```\r\n\r\nHere's the full file result\r\n[differences.txt](https://github.com/pelias/openstreetmap/files/23719/differences.txt)\r\n\r\n\r\n\r\n\r\n"}}