<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Adding tokenization and normalization to collation
Starting point
Step1:  Separate the text of the witness from its inclusion in the collation.
Step2:  Use functions to tokenize the witness text. Start with just one witness, and verify the result by outputting JSON.
Step3:  Add simple normalization. This won’t affect the collation output, but we can verify that it’s working.
Step4:  Change the text to create a more complex example
Step5:  Enhance normalization to recognize that all animals are alike. (This introduces possible complications, which can be addressed through further enhancements.)
Step6:  The animals are now aligned, but the colors aren’t. We can address that through matching
<ASSISTANT_TASK:>
Python Code:
from collatex import *
collation = Collation()
collation.add_plain_witness( "A", "The quick brown fox jumped over the lazy dog.")
collation.add_plain_witness( "B", "The brown fox jumped over the dog." )
collation.add_plain_witness( "C", "The bad fox jumped over the lazy dog." )
table = collate(collation)
print(table)
Explanation: Adding tokenization and normalization to collation
Starting point
End of explanation
collation = Collation()
A_content = "The quick brown fox jumped over the lazy dog."
B_content = "The brown fox jumped over the dog."
C_content = "The bad fox jumped over the lazy dog." 
collation.add_plain_witness( "A", A_content )
collation.add_plain_witness( "B", B_content )
collation.add_plain_witness( "C", C_content )
table = collate(collation)
print(table)
Explanation: Separate the text of the witness from its inclusion in the collation.
End of explanation
import re
def tokenize(input):
    return [create_token(token) for token in re.findall('\S+\s*',input)]
def create_token(input):
    return {"t": input}
collation = Collation()
A_content = "The quick brown fox jumped over the lazy dog."
B_content = "The brown fox jumped over the dog."
C_content = "The bad fox jumped over the lazy dog." 
witness_list = []
witness_list.append({"id": "A", "tokens": tokenize(A_content)})
print(witness_list)
print("\n")
json_input = {"witnesses": witness_list}
print(json_input)
Explanation: Use functions to tokenize the witness text. Start with just one witness, and verify the result by outputting JSON.
End of explanation
import re
def normalize(input):
    return input.lower()
def tokenize(input):
    return [create_token(token) for token in re.findall('\S+\s*',input)]
def create_token(input):
    return {"t": input, "n": normalize(input)}
collation = Collation()
A_content = "The quick brown fox jumped over the lazy dog."
B_content = "The brown fox jumped over the dog."
C_content = "The bad fox jumped over the lazy dog." 
witness_list = []
witness_list.append({"id": "A", "tokens": tokenize(A_content)})
witness_list.append({"id": "B", "tokens": tokenize(B_content)})
witness_list.append({"id": "C", "tokens": tokenize(C_content)})
json_input = {"witnesses": witness_list}
print(json_input)
table = collate(json_input)
print(table)
Explanation: Add simple normalization. This won’t affect the collation output, but we can verify that it’s working.
End of explanation
import re
def normalize(input):
    return input.lower()
def tokenize(input):
    return [create_token(token) for token in re.findall('\S+\s*',input)]
def create_token(input):
    return {"t": input, "n": normalize(input)}
collation = Collation()
A_content = "Look, a gray koala!"
B_content = "Look, a big grey koala!"
C_content = "Look, a big wombat!" 
witness_list = []
witness_list.append({"id": "A", "tokens": tokenize(A_content)})
witness_list.append({"id": "B", "tokens": tokenize(B_content)})
witness_list.append({"id": "C", "tokens": tokenize(C_content)})
json_input = {"witnesses": witness_list}
# print(json_input)
table = collate(json_input)
print(table)
Explanation: Change the text to create a more complex example
End of explanation
import re
import string
def normalize(input):
    # "string.punctuation" returns string of all punctuation marks Python knows
    # the [] are a regex character class
    # this subs all punctuation for an empty space
    input = re.sub('[' + string.punctuation + ']','',input)
    animals = ['koala', 'wombat']
    # animals is a list of all animals
    if input in animals:
        return 'ANIMAL'
    else:
        return input.lower()
def tokenize(input):
    return [create_token(token) for token in re.findall('\S+\s*',input)]
def create_token(input):
    return {"t": input, "n": normalize(input)}
collation = Collation()
A_content = "Look, a gray koala!"
B_content = "Look, a big grey koala!"
C_content = "Look, a big wombat!" 
witness_list = []
witness_list.append({"id": "A", "tokens": tokenize(A_content)})
witness_list.append({"id": "B", "tokens": tokenize(B_content)})
witness_list.append({"id": "C", "tokens": tokenize(C_content)})
json_input = {"witnesses": witness_list}
print(json_input)
table = collate(json_input)
print(table)
Explanation: Enhance normalization to recognize that all animals are alike. (This introduces possible complications, which can be addressed through further enhancements.)
End of explanation
import re
import string
def normalize(input):
    input = re.sub('[' + string.punctuation + ']','',input)
    animals = ['koala', 'wombat']
    if input in animals:
        return 'ANIMAL'
    else:
        return input.lower()
def tokenize(input):
    return [create_token(token) for token in re.findall('\S+\s*',input)]
def create_token(input):
    return {"t": input, "n": normalize(input)}
collation = Collation()
A_content = "Look, a gray koala!"
B_content = "Look, a big grey koala!"
C_content = "Look, a big wombat!" 
witness_list = []
witness_list.append({"id": "A", "tokens": tokenize(A_content)})
witness_list.append({"id": "B", "tokens": tokenize(B_content)})
witness_list.append({"id": "C", "tokens": tokenize(C_content)})
json_input = {"witnesses": witness_list}
# print(json_input)
# CX's near_matching looks for the closest match
table = collate(json_input, near_match=True, segmentation=False)
print(table)
Explanation: The animals are now aligned, but the colors aren’t. We can address that through matching:
End of explanation
<END_TASK>