<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Address Normalization
Step1:  First, let's read the data that we're going to use to normalize and parse the addresses
Step2:  punctuation is the file with the punctuation characters that we want to remove.
prefixes are the the common address prefixes that we want to remove.
stopw are the common Portuguese stopwords that we also want to remove.
Step3:  So what are we doing here? Let's see what normalize_address is doing
Step4:  So we are doing several operations in sequence
Step5:  So how are we doing that?
Step6:  That's the regular expression
Step7:  So now we have our candidates! But the other informations are different from what whe have in our address. Is this a match?
Match
Let's devise a way to match these two addresses
Step8:  We have some prior information about how addresses are and which parts are more important than others. We can devise a matching algorithm with a linear regression, for example, using the knowledge that street names are more important than complements
Step9:  So the similarity of the street name is exactly the same. Let's compare the numbers
Step10:  Ops, still the same. Let's go to complements
Step11:  Ok, there's a small difference, but we will assume that we can work with that! The second canonical address is a better match than the first one
<ASSISTANT_TASK:>
Python Code:
from nltk.corpus import stopwords
import string
from transform.normalizer import *
from transform.parser import *
from match.match import *
import inspect
import jellyfish
from retrieve.search import *
Explanation: Address Normalization
End of explanation
punctuation = set(string.punctuation)
language = 'portuguese'
prefix_file = '../data/prefixes.csv'
with open(prefix_file, 'r') as g:
    prefixes = g.read().splitlines()
address_prefixes = prefixes
stopw = stopwords.words(language)
Explanation: First, let's read the data that we're going to use to normalize and parse the addresses:
End of explanation
address = "Rua XV de Novembro, 123 bloco 23 A"
normalized_address = normalize_address(
    address, punctuation, stopw , address_prefixes)
print("Normalized address: ", normalized_address)
Explanation: punctuation is the file with the punctuation characters that we want to remove.
prefixes are the the common address prefixes that we want to remove.
stopw are the common Portuguese stopwords that we also want to remove.
End of explanation
inspect.getsourcelines(normalize_address)
Explanation: So what are we doing here? Let's see what normalize_address is doing:
End of explanation
parsed_address = parse_address(normalized_address)
print(parsed_address)
Explanation: So we are doing several operations in sequence:
transform_encoding
transform_case
remove_punctuation
remove_stopwords
remove_address_prefixes
Applying the next function to the results of the previous one. So what's next?
Parsing the address
After we normalized the address we want to parse it, selecting the relevant parts. We can do that with Regex or Named Entity Recognition. First, let's try to use regular expressions:
End of explanation
inspect.getsourcelines(parse_address)
Explanation: So how are we doing that?
End of explanation
schema = create_schema()
idx = create_index(schema, 'indexdir')
results = search(parsed_address['street'], 'street', idx)
print(results)
Explanation: That's the regular expression: ^(\\S+\\D*?)\\s*(\\d+)|(\\S.*).
It means:
```
1st Alternative: ^(\S+\D?)\s(\d+)
^ assert position at start of the string
1st Capturing group (\S+\D?)
\S+ match any non-white space character [^\r\n\t\f ]
Quantifier: + Between one and unlimited times, as many times as possible, giving back as needed [greedy]
\D? match any character that's not a digit [^0-9]
Quantifier: ? Between zero and unlimited times, as few times as possible, expanding as needed [lazy]
\s match any white space character [\r\n\t\f ]
Quantifier: * Between zero and unlimited times, as many times as possible, giving back as needed [greedy]
2nd Capturing group (\d+)
\d+ match a digit [0-9]
Quantifier: + Between one and unlimited times, as many times as possible, giving back as needed [greedy]
2nd Alternative: (\S.*)
3rd Capturing group (\S.)
\S match any non-white space character [^\r\n\t\f ]
. matches any character (except newline)
Quantifier: * Between zero and unlimited times, as many times as possible, giving back as needed [greedy]
```
Wow, that's very hard to understand. 
But now we have our address normalized and separated on its components. We can now try to match it with the canonical source. 
Match
The previous steps did not correct for misspellings or other errors. If we have a canonical database, we can try to reduce those errors and transform our address in our database to a canonical form. For that, we have to match the normalized address with our reference database. First, how we calculate that two addresses are similar?
Similarity
We can compute a similarity between two strings. There are several algorithms to do that. We will use the Jaro-Winkler distance for that. There are several others, like 
Cosine Distance
Hamming Distance
Candidates for match
How can we retrieve candidates to match from our canonical database?
There are a few approaches:
Brute force (all against all)
Search by field
We will try to search for candidates and do a match with them.
Search
End of explanation
print(address)
Explanation: So now we have our candidates! But the other informations are different from what whe have in our address. Is this a match?
Match
Let's devise a way to match these two addresses:
End of explanation
similarity(parsed_address['street'],results[0]['street'] )
similarity(parsed_address['street'],results[1]['street'] )
Explanation: We have some prior information about how addresses are and which parts are more important than others. We can devise a matching algorithm with a linear regression, for example, using the knowledge that street names are more important than complements:
End of explanation
similarity(str(parsed_address['number']),str(results[0]['number'] ))
similarity(str(parsed_address['number']),str(results[1]['number'] ))
Explanation: So the similarity of the street name is exactly the same. Let's compare the numbers:
End of explanation
similarity(parsed_address['complement'],results[0]['complement'] )
similarity(parsed_address['complement'],results[1]['complement'] )
Explanation: Ops, still the same. Let's go to complements:
End of explanation
print("Original Address:", address)
print("Canonical Address:", str(results[1]['street']) + ', ' + str(results[1]['number']) + ' ' + str(results[1]['complement']))
Explanation: Ok, there's a small difference, but we will assume that we can work with that! The second canonical address is a better match than the first one:
End of explanation
<END_TASK>