<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Wrangling OpenStreetMap Data
Map area
Step1:  OSM allows a very flexible tagging system, which gives user freedom but causing problem with consistency.
I count the number for all tag in the document.
Step2:  Below I list the top 20 tag keys sorted descending, as the whole list will be too exhaustive to read.
Step3:  Looking through the data I see several things
Step4:  We see for address there are three related tags
Step5:  Address spelling problem
Several street names is actually the same, but spell out differently.
Wikipedia has the name of most of important street in Jakarta. So I will use that as a reference.
To check the similarity between the reference address name with the one we have in our MongoDB, I utilize Python built-in difflib library. SequenceMatcher class can compare and give out ratio of similarity. We are intrested at strings that generate ratio above 0.65 and less then 1. We also ignore the prefix and the suffix like 'raya' which is common suffix for street names in Indonesia.
Step6:  We still need to manually check and replace the street name, but it is a much simpler task.
Incomplete and incorrect city problem
Since I am using the reference from Wikipedia which contain group streets by cities. I can probably update the city information using this reference as well. First let's audit the city information.
Step7:  There are 44 unique different names for city, after looking at the result, many of them are invalid.
Some only have Jakarta as the listed city, what is surprising though is that there are cities from the surroundings area as well. For instance Tangerang, Bekasi and Bogor are included.
Since it is common in Indonesia to use the same street name for multiple city we need to be conservative here. I will only update the city if there is already a city field and it only says 'Jakarta'.
Overuse of addr
<ASSISTANT_TASK:>
Python Code:
from osm_dataauditor import OSMDataAuditor
osm_data = OSMDataAuditor('jakarta_indonesia.osm')
# Basic element check
osm_data.count_element()
Explanation: Wrangling OpenStreetMap Data
Map area: Jakarta, Indonesia
 Data source: https://s3.amazonaws.com/metro-extracts.mapzen.com/jakarta_indonesia.osm.bz2
Overview
After downloading the map data of Jakarta, I do some initial checking on the document manually.
I then run some code to get the feel of the data.
End of explanation
# Check the tag key and element
tag_keys = osm_data.get_tag_keys()
Explanation: OSM allows a very flexible tagging system, which gives user freedom but causing problem with consistency.
I count the number for all tag in the document.
End of explanation
sorted(tag_keys, key=lambda x: x[1], reverse=True)[:20]
Explanation: Below I list the top 20 tag keys sorted descending, as the whole list will be too exhaustive to read.
End of explanation
import re
# Name vs Nama (Nama is Indonesian for Name)
[item for item in tag_keys if re.match('(name|nama)$', item[0], re.I)]
# Province vs propinsi (Propinsi is Indonesian for Province)
[item for item in tag_keys if re.match('(province|propinsi)$', item[0], re.I)]
# Alamat vs address, (Alamat is Indonesian for address)
sorted([item for item in tag_keys if re.match('(addr|alamat)', item[0], re.I)], key=lambda x: x[1], reverse=True)
Explanation: Looking through the data I see several things:
There are only a handful usage of postal code, total of 900. This is not suprising, since in Indonesia we don't use this postal code that much.
There are only a handful entry for telephone, total of 212.
There are quite numerous flood_prone tag, 21000+.
Many of the tags are actually duplicate.
Etc.
On the topic of duplicate tags, below I show several tag that has similar names.
End of explanation
abbreviated_st = osm_data.audit_abbreviated_street_prefixes()
print "Total abbreviated street names:", len(abbreviated_st)
# Print the first 10 rows
abbreviated_st[:10]
abbreviated_alley = osm_data.audit_abbreviated_alley_prefixes()
print "Total abbreviated alley names:", len(abbreviated_alley)
# Print the first 10 rows
abbreviated_alley[:10]
Explanation: We see for address there are three related tags: 'ALAMAT, 'addr:street', 'addr:full'.
Both 'addr:street' and 'addr:full' is valid tag, so we can not merge them.
OSM wiki imply that the using 'addr:street' and other supporting field is better then using 'addr:full' but our data shows that we have more 'addr:full' then 'addr:street' (10038 vs 1441).
Indonesia has a bit complex administrative subdivision. It is divided as follow:
Province
Regency (Kabupaten) or City (Kota)
District (Kecamatan)
Village
For Jakarta (a province), it is divided into 4 cities: Jakarta Selatan, Jakarta Utara, Jakarta Barat and Jakarta Timur.
And then there are non-administrative division like RT and RW. While RT and RW is considered a non-administrative subdivision, it is widely use (The ID card has and requires this information).
I believe this is what lead user to just simply put the whole address in 'addr:full', as this is much simpler. 
Separating it into different bits is often difficult and it does not actually matched OSM address tag (it does not have things like 'Kabupaten' or 'Kecamatan'). But as OSM warns, putting everything in 'addr:full' makes it harder to parse by software.
There are some effort by the community to try to add the division into the data, but the result is not consistent. For instance for Regency we have 'KAB_NAME', 'kab_name', 'Kabupaten', 'kab.', etc. And some way node uses 'admin_level' tag and then put 'kabupaten' in the value.
Problems encountered in Map
Address prefixes problem
The problems with the address prefixes:
Abbreviated street names (Jl. Masjid Almunawarah, Jln Perintis, etc)
Abbreviated alley names (Gg. Kembang)
For the street name there are several variation, that is: 'jl.', 'jln.', 'jl', jln'.
And then some use all upper case, some all lower case and some are mixed.
End of explanation
closely_matched = osm_data.audit_address_similar_names()
closely_matched = sorted(list(closely_matched), key=lambda x: x[2], reverse=True)
print "Total similiar item found: ", len(closely_matched)
# Display top 10 with score
[(reference, found, score) for reference, found, score in closely_matched[:10]]
Explanation: Address spelling problem
Several street names is actually the same, but spell out differently.
Wikipedia has the name of most of important street in Jakarta. So I will use that as a reference.
To check the similarity between the reference address name with the one we have in our MongoDB, I utilize Python built-in difflib library. SequenceMatcher class can compare and give out ratio of similarity. We are intrested at strings that generate ratio above 0.65 and less then 1. We also ignore the prefix and the suffix like 'raya' which is common suffix for street names in Indonesia.
End of explanation
city_names = osm_data.audit_city()
print "Different city names: ", len(city_names)
Explanation: We still need to manually check and replace the street name, but it is a much simpler task.
Incomplete and incorrect city problem
Since I am using the reference from Wikipedia which contain group streets by cities. I can probably update the city information using this reference as well. First let's audit the city information.
End of explanation
from pymongo import MongoClient
client = MongoClient()
db = client['osm_data_import']
# Number of document
db.jakarta.find().count()
# Number of nodes
db.jakarta.find({'type': 'node'}).count()
# Number of way
db.jakarta.find({'type': 'way'}).count()
# Number of unique user
len(db.jakarta.distinct('created.user'))
# Top 10 contributing user
list(db.jakarta.aggregate([{'$group': {'_id': '$created.user', 'count': {'$sum': 1}}}, {'$sort':{'count':-1}}, {'$limit':10}]))
# Place of worship breakdown
list(db.jakarta.aggregate([
        {"$match":{"amenity":{"$exists":1}, "amenity":"place_of_worship"}},
        {"$group":{"_id":"$religion", "count":{"$sum":1}}},
        {"$sort":{"count":-1}}
    ]))
Explanation: There are 44 unique different names for city, after looking at the result, many of them are invalid.
Some only have Jakarta as the listed city, what is surprising though is that there are cities from the surroundings area as well. For instance Tangerang, Bekasi and Bogor are included.
Since it is common in Indonesia to use the same street name for multiple city we need to be conservative here. I will only update the city if there is already a city field and it only says 'Jakarta'.
Overuse of addr:full tag
Unfortunately most of the the address is in the addr:full tag instead of addr:street (4515 vs 461). 
Looking through the content of the addr:full, we can see several variation.
RT RW only no street name, example:
RT 002 RW 08
RT 0014 RW 010
Street name with RT no RW, example:
Jalan sumur batu raya, Rt 07/01
Street name with house number, example:
Jalan Medan Merdeka Barat No. 12
Jalan Kartini 8 dalam no 18
etc.
I won't be cleaning addr:full further (other then the prefixes fix) on this project but I think I need to address this here.
Data Overview
This section contains basic statistics about the dataset and the MongoDB queries used to gather them.
File sizes:
jakarta_indonesia.osm ......... 449.2 MB
jakarta_indonesia.osm.json .... 524.3 MB
End of explanation
<END_TASK>